-- Logs begin at Fri 2022-06-10 12:01:33 UTC, end at Fri 2022-06-10 13:55:31 UTC. -- Jun 10 12:44:12.815803 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jun 10 12:44:12.826979 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: *** Starting uWSGI 2.0.18-debian (64bit) on [Fri Jun 10 12:44:12 2022] *** Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: nodename: dv-cinder-master-3-iscsi Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: machine: x86_64 Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: clock source: unix Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: pcre jit disabled Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: detected number of CPU cores: 4 Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: current working directory: / Jun 10 12:44:12.829106 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: detected binary path: /usr/bin/uwsgi-core Jun 10 12:44:12.829773 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: your processes number limit is 47680 Jun 10 12:44:12.829773 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: your memory page size is 4096 bytes Jun 10 12:44:12.829773 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: detected max file descriptor number: 2048 Jun 10 12:44:12.829773 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: lock engine: pthread robust mutexes Jun 10 12:44:12.829773 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: thunder lock: enabled Jun 10 12:44:12.829773 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jun 10 12:44:12.829773 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] Jun 10 12:44:12.874043 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: Python main interpreter initialized at 0x555f6c87d9c0 Jun 10 12:44:12.874043 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: python threads support enabled Jun 10 12:44:12.874043 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: your server socket listen backlog is limited to 100 connections Jun 10 12:44:12.874043 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: your mercy for graceful operations on workers is 90 seconds Jun 10 12:44:12.874425 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: mapped 403077 bytes (393 KB) for 2 cores Jun 10 12:44:12.874501 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: *** Operational MODE: preforking *** Jun 10 12:44:12.874772 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: *** uWSGI is running in multiple interpreter mode *** Jun 10 12:44:12.874772 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: spawned uWSGI master process (pid: 80288) Jun 10 12:44:12.874983 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jun 10 12:44:12.876101 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: spawned uWSGI worker 1 (pid: 80299, cores: 1) Jun 10 12:44:12.876101 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: spawned uWSGI worker 2 (pid: 80300, cores: 1) Jun 10 12:44:12.876101 dv-cinder-master-3-iscsi devstack@keystone.service[80288]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jun 10 12:44:18.433176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jun 10 12:44:18.433176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: from cryptography.utils import int_from_bytes Jun 10 12:44:18.433881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jun 10 12:44:18.433881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: from cryptography.utils import int_from_bytes Jun 10 12:44:18.439315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jun 10 12:44:18.439315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: from cryptography.utils import int_from_bytes Jun 10 12:44:18.439772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jun 10 12:44:18.439772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: from cryptography.utils import int_from_bytes Jun 10 12:44:18.481907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jun 10 12:44:18.482117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jun 10 12:44:18.482349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] command line args: [] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jun 10 12:44:18.482622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jun 10 12:44:18.482802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ================================================================================ {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jun 10 12:44:18.483057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] admin_token = **** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.483305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] config_dir = [] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.483492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.483676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] config_source = [] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.483854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] debug = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.484053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.485442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.485442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.485442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.485442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.485684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jun 10 12:44:18.485763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] list_limit = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.485830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jun 10 12:44:18.485901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] log_config_append = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.485967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] command line args: [] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jun 10 12:44:18.486025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] log_dir = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jun 10 12:44:18.486198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] log_file = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ================================================================================ {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jun 10 12:44:18.486317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] admin_token = **** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] config_dir = [] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486812 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] config_source = [] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.486965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.487048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] debug = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.487120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.487239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.487463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.487529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.487599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.487664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.487748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] list_limit = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] log_config_append = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488643 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.488893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] log_dir = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.489020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.489141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] log_file = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.489261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.489374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.489493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.489620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.489740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.489882 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] publish_errors = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.490988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.491108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.491226 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.491410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.491410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.491624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] standard_threads = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.491742 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.491854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.491972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.492103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.492103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.492326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.492439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] use_journal = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.492572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.492696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] use_json = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.492696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] use_stderr = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.492696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] use_syslog = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.493042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.25/identity {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.493170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.493285 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] publish_errors = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.493397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.493509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.493628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.493751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.493865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.493976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.494088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.494201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.494201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.494487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.494610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] standard_threads = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.494724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.494843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.494957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.495076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.495200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.495314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.495426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.495538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] use_journal = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.495538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] use_json = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.495757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.495878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] use_stderr = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.495997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.496116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] use_syslog = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.496237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.496357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jun 10 12:44:18.496470 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.496591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.496719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.496861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.496977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.497089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.497208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.497333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.497447 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.497565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.497679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.498282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.498404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.498529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.498644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.498762 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.498875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.498987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.499099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.499219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.499339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.499453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.499453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.499663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.499775 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.499898 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.500019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.500141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.500258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.500371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.500371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.500667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.500806 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.500924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.501043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.501156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.501280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.501395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.501513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.501627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.501738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.501849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.501970 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.502083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.502200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.502313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.502432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.502552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.502674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.502788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.502900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503011 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.503981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.504077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.504170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.504263 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.504394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.504500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.504657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.504805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.504924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.505039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.505163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.505276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.505398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.505514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.505634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.505749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.505861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.505979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.506101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.506206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.506329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.506444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.506559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.506672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.506785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.506907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.507023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.507136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.507249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.507361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.507474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.507599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.507715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.507825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.507938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.508051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.508163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.508290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.508405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.508508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.508619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.508741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.508890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.509020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.509020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.509234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.509234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.509455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.509569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.509693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.509915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.510030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.510142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.510255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.510376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.510492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.510492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.510492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.510492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.510492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.511059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.511059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.511284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.511397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.511516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.511636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.511636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.511929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.512044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.512164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.512279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.512391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.512512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.512627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.512740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.512919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.513036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.513164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.513290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.513453 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.514109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.514231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.514346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.514460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.514573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.514687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.514810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.514926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.515041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.515156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.515270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.515393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.515510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.515630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.515630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.515855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.516039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.516280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.516520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.516778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.516910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.517028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.517149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.517274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.517390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.517505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.517619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.517733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.517854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.517979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.518095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.518210 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.518324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.518438 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.518552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.518674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.518790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.518912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.519027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.519147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.519264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.519387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.519503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.519623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.519747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.519864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] auth.external = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.519985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.520110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.520228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.520343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.520458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.520571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.520686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.520825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] auth.password = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.520944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.521059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] auth.token = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.521173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.521287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.521428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.521574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.521722 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.521852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.521970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.522083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.522197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.522311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.522426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] auth.external = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.522549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.522665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.522779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.522893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.523009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] credential.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.523122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.523244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.523361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] auth.password = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.523482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.523599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] auth.token = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.523713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.523833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.523960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.524077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.524191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.524305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.524425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.524542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.524665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.524825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.524944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.525059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.525173 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.525288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.525288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] credential.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.525520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.525635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.525756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.525886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.526009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.526134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.526250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.526377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.526493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.526608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.526722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.526912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.527031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] federation.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.527146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.527260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.527374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.527487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.527613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.527731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.527846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.527960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.528074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.528188 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.528311 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.528434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.528556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.528671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.528801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.528931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.529064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.529181 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.529296 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] federation.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.529410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.529524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.529647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.529765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.529880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.529994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.530107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.530221 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.530344 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.530460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.530575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.530940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.531056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.531171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.531294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.531411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.531525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.531640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.531754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.531870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.531993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.532110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.532224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.532338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.532451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.532565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.532688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.532829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.532959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.533077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.533192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.533562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.533693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.533811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.533926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.534042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.534156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.534271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.534398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.534516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.534631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.534745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.534859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.534974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.535098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.535215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.535330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.535445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.535559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.535672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.535796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.535912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.536027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.536142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.536256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.536375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.536499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.536616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.536731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.536871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.536989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.537105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.537229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.537346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.537461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.537575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.537689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.537803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.537927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.538044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.538160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.538274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.538388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.538511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.538628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.538743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.538857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.538972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.539086 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.539211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.539327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.539442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.539556 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.539671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.539785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.539908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.540024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.540138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.540254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.540368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.540481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.540604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.540721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.540889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.541006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.541121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.541235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.541364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.541482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.541597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.541711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.541826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.541940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.542067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.542190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.542306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.542422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.542536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.542650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.542778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.542896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.543011 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.543125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.543240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.543354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.543478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.543595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.543710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.543825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.543939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.544053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.544177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.544293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.544409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.544524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.544639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.544779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.544912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.545030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.545145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.545266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.545383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.545498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.545621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.545738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.545853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.545967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.546081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.546195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.546322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.546322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.546542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.546656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.546777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.546893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.547016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.547132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.547246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.547360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.547480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.547597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.547721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.547838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.547952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.548065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.548186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.548302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.548425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.548541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.548657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.548824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.548943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.549058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.549186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.549303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.549418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.549532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.549647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.549762 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.549950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.550069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.550184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.550299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.550413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.550535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.550660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.550778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.550893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.551007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.551121 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.551236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.551359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] resource.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.551475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.551590 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.551705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.551827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.551944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.552067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.552185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.552306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.552423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.552538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.552652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.552794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.552914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.553030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.553144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.553259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.553374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] resource.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.553501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.553620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.553734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] role.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.553854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.553971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] role.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.554094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.554211 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.554326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.554440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.554669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.554792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.554909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555607 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555722 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] role.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] role.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.555951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.556175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.556175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.556397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.556511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.556625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.556625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.556869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.556987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.557103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.557217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.557332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.557446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.557569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.557685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.557799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.557913 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.558026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.558141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.558263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.558379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.558494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.558608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.558729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.558845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.558968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.559084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.559234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.559234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.559452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.559575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.559693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.559808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.559923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.560037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.560151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.560796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.560919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.561035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.561150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.561265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.561380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.561508 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.561626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.561741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.561855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.561969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.562090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.562215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.562332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.562447 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] token.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.562561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.562676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.562790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.562913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.563030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.563145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.563250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.563364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.563479 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.563602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.563718 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.563832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.563946 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.564060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.564175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.564297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.564414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] token.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.564529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.564643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.564757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.564888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.565016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.565134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.565249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.565364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.565479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.565594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.565716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.565833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.565947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.566061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.566175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.566292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.566416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.566540 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.566656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.connection = **** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.566780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.566780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.567957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.568176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.568296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.568296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.568511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.568635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.568635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.568899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.connection = **** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.569028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.569217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.569335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.569335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.569551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.569676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.569793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.569908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.569908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.570123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.570237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.570237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.570463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.570578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80299) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jun 10 12:44:18.570694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.570694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.570918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.571174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.571533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.571903 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.572261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.572639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.573027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.573418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.573798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.574192 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.574574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jun 10 12:44:18.574825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80300) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jun 10 12:44:18.687443 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.687614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.687819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.696323 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.703163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.703395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.703614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.712209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.725410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.725410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.730570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.730796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.778413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.782169 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.827377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.827609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.828287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.828525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.892490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.892715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.892963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.893116 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.893200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.893275 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.893385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.893469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.893601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.893694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.893842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.894061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.916681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.916810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.916963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.917046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.917152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.917237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.917373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.917471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.917598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.917681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.917813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.917895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.918028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.918112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.929626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.935046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.943013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.943247 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.943474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.943695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.943914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.944130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.944348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80299) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.947333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.947572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.947793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.948009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.948220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.948432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.948644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80300) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jun 10 12:44:18.961469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.961569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.961681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jun 10 12:44:18.961766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jun 10 12:44:18.962020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.962105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.962224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jun 10 12:44:18.962306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jun 10 12:44:18.962660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.962751 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.962920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.963003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.963176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.963257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.963427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.963517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.974715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.974809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.974990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.975073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.975255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.975338 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.975528 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.975612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.984133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:18.984241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:18.987500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:18.987588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:18.987781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.987863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.988055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.988150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.988330 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.988413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.988606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.988688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.988885 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.988969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.989149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.989232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.989415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.989507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.989674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.989754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.989938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.990018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.990196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:18.990285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.016502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.016596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.018136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.018252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.018412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.018528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.023625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.023817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.023900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.024099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.024183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.024382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.032295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.032571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.032654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.032873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.037040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.037320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.037416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.037599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.037688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.037875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.037957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.038145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.038550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.038550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.038710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.038710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.038842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.038914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.038986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.039183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.057458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.057714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.057795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.057990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.062791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.063028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.066436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.066652 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.066736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.066945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.071807 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.071983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.075424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.075564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.079306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.079402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.079605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.079688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.079886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.079983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.080163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.080247 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.080433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.080527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.080712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.080809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.100185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.100283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.100465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.100550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.100731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.100842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.101012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.101093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.108901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.108993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.110594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.110677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.110863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.110959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.113911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.114005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.114183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.114267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.115732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.115838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.116024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.116108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.116302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.116384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.129973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.130097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.130249 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.130375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.130523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.130653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.130797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.130929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.131077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.131213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.131346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.131484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.131623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.131754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.149444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.149622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.152676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.152887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.156732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.156946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.158356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.158555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.160107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.160317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.160401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.160594 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.160685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.160905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.167856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.168134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.169486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.169828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.172731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.173040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.173131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.173320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.173416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.173600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.173683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.173869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.182735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.183019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.183104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.183295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.183377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.183574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.191586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80300) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.191859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.191944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80299) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jun 10 12:44:19.192145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.192228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.192413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.192505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.192686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.192790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.192981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.193102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.193248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.193334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.193517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.193598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.193784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.193863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.194044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.194153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.194409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.194490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.194681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.194770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.194944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.195026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.195203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80300) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.195284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.195461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80299) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jun 10 12:44:19.233401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WSGI app 0 (mountpoint='') ready in 7 seconds on interpreter 0x555f6c87d9c0 pid: 80299 (default app) Jun 10 12:44:19.233888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WSGI app 0 (mountpoint='') ready in 7 seconds on interpreter 0x555f6c87d9c0 pid: 80300 (default app) Jun 10 12:44:19.236821 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81793a5-3d61-40d4-ac43-b1c863505888 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:19.237119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81793a5-3d61-40d4-ac43-b1c863505888 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:19.237398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81793a5-3d61-40d4-ac43-b1c863505888 None None] PATH_INFO: `/v3` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:19.241628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1/1] 10.222.0.25 () {50 vars in 791 bytes} [Fri Jun 10 12:44:19 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:27.948711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ce58de-7fce-443e-a059-b13e43f23f88 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:27.949297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ce58de-7fce-443e-a059-b13e43f23f88 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:27.949297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ce58de-7fce-443e-a059-b13e43f23f88 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:27.954212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1/2] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:27 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:44:27.964951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:27.964951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:27.964951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:28.031615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: INFO dbcounter [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] Registered counter for database keystone Jun 10 12:44:28.052798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG oslo_db.sqlalchemy.engines [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80299) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jun 10 12:44:28.058976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writer thread running {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jun 10 12:44:28.237833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG passlib.handlers.bcrypt [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80299) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jun 10 12:44:28.238156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG passlib.handlers.bcrypt [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80299) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jun 10 12:44:28.291780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:28.293397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:28.424041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 2/3] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:27 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 466 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jun 10 12:44:28.461757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ce58de-7fce-443e-a059-b13e43f23f88 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:28.531985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: INFO dbcounter [None req-66ce58de-7fce-443e-a059-b13e43f23f88 None None] Registered counter for database keystone Jun 10 12:44:28.550724 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG oslo_db.sqlalchemy.engines [None req-66ce58de-7fce-443e-a059-b13e43f23f88 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80300) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jun 10 12:44:28.556952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writer thread running {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jun 10 12:44:28.676869 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ce58de-7fce-443e-a059-b13e43f23f88 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:28.681157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30377623-8d49-4522-8932-0ec29f8cc0a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:28.681810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30377623-8d49-4522-8932-0ec29f8cc0a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:28.682100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30377623-8d49-4522-8932-0ec29f8cc0a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:28.682380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30377623-8d49-4522-8932-0ec29f8cc0a2 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:28.690523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30377623-8d49-4522-8932-0ec29f8cc0a2 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:28.743813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WARNING keystone.common.rbac_enforcer.enforcer [None req-30377623-8d49-4522-8932-0ec29f8cc0a2 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jun 10 12:44:28.744160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30377623-8d49-4522-8932-0ec29f8cc0a2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:28.755072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 2/4] 10.222.0.25 () {58 vars in 1324 bytes} [Fri Jun 10 12:44:28 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 12:44:28.761135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:28.778328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a3c736-7cba-4998-8bf1-fd44d44be14f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:28.783188 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7267eb5-abc2-4ee0-8a76-c704fdb489d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:28.784237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7267eb5-abc2-4ee0-8a76-c704fdb489d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:28.784529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7267eb5-abc2-4ee0-8a76-c704fdb489d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:28.784837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7267eb5-abc2-4ee0-8a76-c704fdb489d4 admin admin] PATH_INFO: `/v3/projects/96340edb86a840a88d51d74872a27bac` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:28.786340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7267eb5-abc2-4ee0-8a76-c704fdb489d4 admin admin] RBAC: Authorizing `identity:get_project(project_id=96340edb86a840a88d51d74872a27bac)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:28.837095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING keystone.common.rbac_enforcer.enforcer [None req-e7267eb5-abc2-4ee0-8a76-c704fdb489d4 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jun 10 12:44:28.837444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7267eb5-abc2-4ee0-8a76-c704fdb489d4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:28.844458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 3/5] 10.222.0.25 () {56 vars in 1207 bytes} [Fri Jun 10 12:44:28 2022] GET /identity/v3/projects/96340edb86a840a88d51d74872a27bac => generated 344 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:30.902961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca471b34-b1c5-4fda-8227-1c731e4ffb90 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:30.903505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca471b34-b1c5-4fda-8227-1c731e4ffb90 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:30.903505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca471b34-b1c5-4fda-8227-1c731e4ffb90 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:30.907934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 3/6] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:44:30.913325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47afdf82-544c-4270-97da-921d1b7bcf8b None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:30.913736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47afdf82-544c-4270-97da-921d1b7bcf8b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:30.913899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47afdf82-544c-4270-97da-921d1b7bcf8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:30.932817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-47afdf82-544c-4270-97da-921d1b7bcf8b None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:30.934352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-47afdf82-544c-4270-97da-921d1b7bcf8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:30.942828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 4/7] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:30 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jun 10 12:44:30.964452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca471b34-b1c5-4fda-8227-1c731e4ffb90 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:30.976718 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca471b34-b1c5-4fda-8227-1c731e4ffb90 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:30.980843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31a9f84-f3d7-4565-8195-8670293dba6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:30.981467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a9f84-f3d7-4565-8195-8670293dba6e admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:30.982306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a9f84-f3d7-4565-8195-8670293dba6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:30.982586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a9f84-f3d7-4565-8195-8670293dba6e admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:30.990111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31a9f84-f3d7-4565-8195-8670293dba6e admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:30.991877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31a9f84-f3d7-4565-8195-8670293dba6e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:31.000384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 4/8] 10.222.0.25 () {58 vars in 1324 bytes} [Fri Jun 10 12:44:30 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 12:44:31.005748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47afdf82-544c-4270-97da-921d1b7bcf8b None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:31.014516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47afdf82-544c-4270-97da-921d1b7bcf8b None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:31.018883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3396ba1e-3545-44b9-93f1-29fa65f8b11d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:31.019745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3396ba1e-3545-44b9-93f1-29fa65f8b11d admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:31.020032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3396ba1e-3545-44b9-93f1-29fa65f8b11d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:31.020312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3396ba1e-3545-44b9-93f1-29fa65f8b11d admin admin] PATH_INFO: `/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:31.021743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3396ba1e-3545-44b9-93f1-29fa65f8b11d admin admin] RBAC: Authorizing `identity:get_user(user_id=05ecde7b5ffd440ea8bdbb9c0c6418c1)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:31.023599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3396ba1e-3545-44b9-93f1-29fa65f8b11d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:31.025193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 5/9] 10.222.0.25 () {56 vars in 1198 bytes} [Fri Jun 10 12:44:31 2022] GET /identity/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1 => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:33.178569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9015c22d-86f5-431f-bb2a-19ea7a8c9844 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.179095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9015c22d-86f5-431f-bb2a-19ea7a8c9844 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.179179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9015c22d-86f5-431f-bb2a-19ea7a8c9844 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.180467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 5/10] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:33.185646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527975fb-7b34-454f-9c19-19fb6b2c38fc None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.186036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527975fb-7b34-454f-9c19-19fb6b2c38fc None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.186219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-527975fb-7b34-454f-9c19-19fb6b2c38fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.198130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e336d5-664d-4a3e-8f85-cbe240984b1c None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.198417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e336d5-664d-4a3e-8f85-cbe240984b1c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.198697 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e336d5-664d-4a3e-8f85-cbe240984b1c None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.199996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 6/11] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:44:33.203154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-527975fb-7b34-454f-9c19-19fb6b2c38fc None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:33.204357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-527975fb-7b34-454f-9c19-19fb6b2c38fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:33.205301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a5b881-8d66-4e25-a884-70609e836e7f None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.205592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a5b881-8d66-4e25-a884-70609e836e7f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.205873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a5b881-8d66-4e25-a884-70609e836e7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.244015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG passlib.handlers.bcrypt [None req-97a5b881-8d66-4e25-a884-70609e836e7f None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80300) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jun 10 12:44:33.244444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG passlib.handlers.bcrypt [None req-97a5b881-8d66-4e25-a884-70609e836e7f None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80300) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jun 10 12:44:33.288326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 6/12] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:33.292167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-97a5b881-8d66-4e25-a884-70609e836e7f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:33.293784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-97a5b881-8d66-4e25-a884-70609e836e7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:33.311832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527975fb-7b34-454f-9c19-19fb6b2c38fc None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:33.334954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:33.396179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 7/13] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:44:33.414822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-527975fb-7b34-454f-9c19-19fb6b2c38fc None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:33.419453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a5b881-8d66-4e25-a884-70609e836e7f None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:33.449493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad5316b-3624-4fe6-806e-6cbc468b80cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:33.450848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad5316b-3624-4fe6-806e-6cbc468b80cd None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.451130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad5316b-3624-4fe6-806e-6cbc468b80cd None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.451408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad5316b-3624-4fe6-806e-6cbc468b80cd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.462336 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad5316b-3624-4fe6-806e-6cbc468b80cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-dad5316b-3624-4fe6-806e-6cbc468b80cd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:33.473416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:33.474490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:33.475559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 7/14] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:44:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:44:33.480236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad5316b-3624-4fe6-806e-6cbc468b80cd None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:33.521288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a5b881-8d66-4e25-a884-70609e836e7f None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:33.523612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:33.562194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793f39ee-b8f0-4519-97e5-6cdaeec89bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:33.563102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793f39ee-b8f0-4519-97e5-6cdaeec89bd4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.563395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793f39ee-b8f0-4519-97e5-6cdaeec89bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.563675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-793f39ee-b8f0-4519-97e5-6cdaeec89bd4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.567274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:33.588056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad5316b-3624-4fe6-806e-6cbc468b80cd None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:33.623229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e021a6a-3f58-4a92-ba90-00e44535c51d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:33.623889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e021a6a-3f58-4a92-ba90-00e44535c51d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.624169 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e021a6a-3f58-4a92-ba90-00e44535c51d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.624445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e021a6a-3f58-4a92-ba90-00e44535c51d None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.625234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e021a6a-3f58-4a92-ba90-00e44535c51d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:33.625541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e021a6a-3f58-4a92-ba90-00e44535c51d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:33.626945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e021a6a-3f58-4a92-ba90-00e44535c51d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:33.638560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 8/15] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:33.644078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e021a6a-3f58-4a92-ba90-00e44535c51d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:33.651494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793f39ee-b8f0-4519-97e5-6cdaeec89bd4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:33.653084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-793f39ee-b8f0-4519-97e5-6cdaeec89bd4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:33.741497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e021a6a-3f58-4a92-ba90-00e44535c51d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:33.748271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 8/16] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:44:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jun 10 12:44:33.753868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793f39ee-b8f0-4519-97e5-6cdaeec89bd4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:33.756509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:33.775267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fce8f5-3105-47c4-b1e5-70c7d6323b76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:33.783686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fce8f5-3105-47c4-b1e5-70c7d6323b76 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.783686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fce8f5-3105-47c4-b1e5-70c7d6323b76 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.783686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fce8f5-3105-47c4-b1e5-70c7d6323b76 None admin] PATH_INFO: `/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.783686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fce8f5-3105-47c4-b1e5-70c7d6323b76 None admin] RBAC: Authorizing `identity:get_user(user_id=05ecde7b5ffd440ea8bdbb9c0c6418c1)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:33.783686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fce8f5-3105-47c4-b1e5-70c7d6323b76 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:33.783686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 9/17] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:33 2022] GET /identity/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1 => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:33.786536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fce8f5-3105-47c4-b1e5-70c7d6323b76 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:33.800746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:33.850787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-793f39ee-b8f0-4519-97e5-6cdaeec89bd4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:33.883603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bcd512-3a6c-4663-bb1b-53bfdaeda7b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:33.884383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bcd512-3a6c-4663-bb1b-53bfdaeda7b8 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.884667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bcd512-3a6c-4663-bb1b-53bfdaeda7b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.884970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4bcd512-3a6c-4663-bb1b-53bfdaeda7b8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.885270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fce8f5-3105-47c4-b1e5-70c7d6323b76 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:33.903665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bcd512-3a6c-4663-bb1b-53bfdaeda7b8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:33.905429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4bcd512-3a6c-4663-bb1b-53bfdaeda7b8 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:33.918537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f99349-89cb-44ec-90bf-7c902d947ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:33.919846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f99349-89cb-44ec-90bf-7c902d947ca4 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:33.920127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f99349-89cb-44ec-90bf-7c902d947ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:33.920399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f99349-89cb-44ec-90bf-7c902d947ca4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:33.921653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f99349-89cb-44ec-90bf-7c902d947ca4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:33.923102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f99349-89cb-44ec-90bf-7c902d947ca4 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:33.924706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 10/18] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:33 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c4bcd512-3a6c-4663-bb1b-53bfdaeda7b8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:44:33.929641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:33.930638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:33.931496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:44:33.931496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:44:33.931496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:44:33.931496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:44:33.931694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f99349-89cb-44ec-90bf-7c902d947ca4 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:33.931772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 9/19] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:44:33.940758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bcd512-3a6c-4663-bb1b-53bfdaeda7b8 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:33.999770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:34.017803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:34.036646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f99349-89cb-44ec-90bf-7c902d947ca4 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:34.041635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4bcd512-3a6c-4663-bb1b-53bfdaeda7b8 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:34.070082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e25944b-4db1-48c0-832e-ed0a604fd1a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:34.070730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e25944b-4db1-48c0-832e-ed0a604fd1a0 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:34.071013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e25944b-4db1-48c0-832e-ed0a604fd1a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:34.071294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e25944b-4db1-48c0-832e-ed0a604fd1a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:34.072122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e25944b-4db1-48c0-832e-ed0a604fd1a0 None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.domain.id=default, user.id=05ecde7b5ffd440ea8bdbb9c0c6418c1) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:34.072432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e25944b-4db1-48c0-832e-ed0a604fd1a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:34.073915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e25944b-4db1-48c0-832e-ed0a604fd1a0 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:34.075637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2f2bb84-9b69-4031-ba90-5609b45d3e85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:34.076293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f2bb84-9b69-4031-ba90-5609b45d3e85 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:34.076575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f2bb84-9b69-4031-ba90-5609b45d3e85 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:34.076880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2f2bb84-9b69-4031-ba90-5609b45d3e85 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:34.077617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f2bb84-9b69-4031-ba90-5609b45d3e85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:34.077930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f2bb84-9b69-4031-ba90-5609b45d3e85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:34.079308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2f2bb84-9b69-4031-ba90-5609b45d3e85 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:34.080792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 11/20] 10.222.0.25 () {56 vars in 1322 bytes} [Fri Jun 10 12:44:33 2022] GET /identity/v3/role_assignments?user.id=05ecde7b5ffd440ea8bdbb9c0c6418c1&scope.domain.id=default&role.id=766799d1b6c049168590571f1a959b3b => generated 233 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:34.098772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 10/21] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:44:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:36.420083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a86829-d501-433e-93bf-a887dc4a3726 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:36.420636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a86829-d501-433e-93bf-a887dc4a3726 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:36.420636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a86829-d501-433e-93bf-a887dc4a3726 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:36.422292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 12/22] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:36.427702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4807c31-a450-4ee2-bc84-a3c74abda35c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:36.428094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4807c31-a450-4ee2-bc84-a3c74abda35c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:36.428258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4807c31-a450-4ee2-bc84-a3c74abda35c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:36.453801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a4807c31-a450-4ee2-bc84-a3c74abda35c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:36.455007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a4807c31-a450-4ee2-bc84-a3c74abda35c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:36.541532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 11/23] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:44:36.575604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a86829-d501-433e-93bf-a887dc4a3726 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:36.680837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e80d61-2560-4226-a308-cd2cb5d62a73 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:36.681192 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e80d61-2560-4226-a308-cd2cb5d62a73 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:36.681616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e80d61-2560-4226-a308-cd2cb5d62a73 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:36.684073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 12/24] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:36.689931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b79328-f94d-4e25-aeec-84c2a034235c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:36.690365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b79328-f94d-4e25-aeec-84c2a034235c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:36.690820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b79328-f94d-4e25-aeec-84c2a034235c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:36.722557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-59b79328-f94d-4e25-aeec-84c2a034235c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:36.723758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-59b79328-f94d-4e25-aeec-84c2a034235c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:36.780572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a86829-d501-433e-93bf-a887dc4a3726 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:36.823876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:36.865146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f0593-c211-4147-9118-26c8cea9ab32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:36.865958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f0593-c211-4147-9118-26c8cea9ab32 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:36.866239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f0593-c211-4147-9118-26c8cea9ab32 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:36.866512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4f0593-c211-4147-9118-26c8cea9ab32 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:36.876977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 13/25] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:36.890615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4f0593-c211-4147-9118-26c8cea9ab32 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-cb4f0593-c211-4147-9118-26c8cea9ab32 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:36.902391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:36.903290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:36.904043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:36.904609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 13/26] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:44:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:36.910339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b79328-f94d-4e25-aeec-84c2a034235c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:36.930011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f0593-c211-4147-9118-26c8cea9ab32 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:36.936524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:37.133410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b79328-f94d-4e25-aeec-84c2a034235c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:37.217503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4f0593-c211-4147-9118-26c8cea9ab32 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:37.223377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50332838-b81c-4708-9ff1-78a2d515e761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:37.225304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50332838-b81c-4708-9ff1-78a2d515e761 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.225790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50332838-b81c-4708-9ff1-78a2d515e761 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.230704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50332838-b81c-4708-9ff1-78a2d515e761 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.231781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50332838-b81c-4708-9ff1-78a2d515e761 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:37.232292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50332838-b81c-4708-9ff1-78a2d515e761 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:37.233958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50332838-b81c-4708-9ff1-78a2d515e761 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:37.252572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 14/27] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 343 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:37.256410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acd31a-378a-4543-a9e3-7312508a6cd9 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.257762 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acd31a-378a-4543-a9e3-7312508a6cd9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.258242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95acd31a-378a-4543-a9e3-7312508a6cd9 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.263639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 15/28] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:37 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:37.263639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95acd31a-378a-4543-a9e3-7312508a6cd9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:37.276394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d195de5-ca77-48b1-a969-7d7b0516da7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:37.277524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d195de5-ca77-48b1-a969-7d7b0516da7a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.278517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d195de5-ca77-48b1-a969-7d7b0516da7a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.279017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d195de5-ca77-48b1-a969-7d7b0516da7a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.290214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d195de5-ca77-48b1-a969-7d7b0516da7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:37.291968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d195de5-ca77-48b1-a969-7d7b0516da7a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:37.309182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-7d195de5-ca77-48b1-a969-7d7b0516da7a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:44:37.312526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:37.313341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:37.314149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:44:37.314149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:44:37.314149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:44:37.314149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:37.316076 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 14/29] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 387 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:37.317571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3ce6c2-917f-4513-bbfa-eb77221c5ed5 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.318020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3ce6c2-917f-4513-bbfa-eb77221c5ed5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.322084 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3ce6c2-917f-4513-bbfa-eb77221c5ed5 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.326125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 15/30] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:37 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:37.327635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579690c1-1017-453a-af43-5881caa0620a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.328128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579690c1-1017-453a-af43-5881caa0620a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.328654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579690c1-1017-453a-af43-5881caa0620a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.350310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-579690c1-1017-453a-af43-5881caa0620a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:37.351841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-579690c1-1017-453a-af43-5881caa0620a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:37.384722 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95acd31a-378a-4543-a9e3-7312508a6cd9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:37.405750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:37.422159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5041ce-1613-41f9-b989-ceae1f35411e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:37.423269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5041ce-1613-41f9-b989-ceae1f35411e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.423708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5041ce-1613-41f9-b989-ceae1f35411e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.424132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5041ce-1613-41f9-b989-ceae1f35411e None admin] PATH_INFO: `/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.424983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 16/31] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:37.425964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579690c1-1017-453a-af43-5881caa0620a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:37.426660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5041ce-1613-41f9-b989-ceae1f35411e None admin] RBAC: Authorizing `identity:get_user(user_id=05ecde7b5ffd440ea8bdbb9c0c6418c1)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:37.428379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5041ce-1613-41f9-b989-ceae1f35411e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:37.430641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 16/32] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:37 2022] GET /identity/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1 => generated 247 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:37.432049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1db5a0-219c-44b4-9c46-e3717e1b72e4 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.432498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1db5a0-219c-44b4-9c46-e3717e1b72e4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.432960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe1db5a0-219c-44b4-9c46-e3717e1b72e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.451228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-fe1db5a0-219c-44b4-9c46-e3717e1b72e4 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:37.452636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-fe1db5a0-219c-44b4-9c46-e3717e1b72e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:37.526490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 17/33] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:37.527359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1db5a0-219c-44b4-9c46-e3717e1b72e4 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:37.533167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579690c1-1017-453a-af43-5881caa0620a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:37.563927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:37.569117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46f52c2-e15d-4ebf-860f-c319b9155af1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:37.569792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46f52c2-e15d-4ebf-860f-c319b9155af1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.570078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46f52c2-e15d-4ebf-860f-c319b9155af1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.570362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a46f52c2-e15d-4ebf-860f-c319b9155af1 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.571094 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46f52c2-e15d-4ebf-860f-c319b9155af1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:37.571403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46f52c2-e15d-4ebf-860f-c319b9155af1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:37.572799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a46f52c2-e15d-4ebf-860f-c319b9155af1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:37.591247 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 17/34] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:44:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:37.592072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46f52c2-e15d-4ebf-860f-c319b9155af1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:37.648790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:37.652856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe1db5a0-219c-44b4-9c46-e3717e1b72e4 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:37.693912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ac2d63-c041-4879-8376-3afc25edc5b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:37.694933 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ac2d63-c041-4879-8376-3afc25edc5b0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.695222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ac2d63-c041-4879-8376-3afc25edc5b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.695503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ac2d63-c041-4879-8376-3afc25edc5b0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.696711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ac2d63-c041-4879-8376-3afc25edc5b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:37.698377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ac2d63-c041-4879-8376-3afc25edc5b0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:37.700939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 18/35] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:37.701590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ac2d63-c041-4879-8376-3afc25edc5b0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:37.712484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a46f52c2-e15d-4ebf-860f-c319b9155af1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:37.758114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9bb601-e070-41bd-856d-a495fa45fedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:37.758802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9bb601-e070-41bd-856d-a495fa45fedf None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.759603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9bb601-e070-41bd-856d-a495fa45fedf None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.759897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9bb601-e070-41bd-856d-a495fa45fedf None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.760946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9bb601-e070-41bd-856d-a495fa45fedf None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:37.763100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9bb601-e070-41bd-856d-a495fa45fedf None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:37.811868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ac2d63-c041-4879-8376-3afc25edc5b0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:37.815002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:37.846484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08487141-f545-4925-a010-274ab48e2508 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:37.847164 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08487141-f545-4925-a010-274ab48e2508 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:37.847439 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08487141-f545-4925-a010-274ab48e2508 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:37.847720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08487141-f545-4925-a010-274ab48e2508 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:37.848507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08487141-f545-4925-a010-274ab48e2508 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:37.849931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08487141-f545-4925-a010-274ab48e2508 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:38.093630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 18/36] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Jun 10 12:44:37 2022] POST /identity/v3/roles => generated 225 bytes in 502 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:44:38.094892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9bb601-e070-41bd-856d-a495fa45fedf None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:38.141342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 19/37] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Jun 10 12:44:37 2022] POST /identity/v3/roles => generated 219 bytes in 440 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:44:38.142790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08487141-f545-4925-a010-274ab48e2508 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:38.218053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9bb601-e070-41bd-856d-a495fa45fedf None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:38.228108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:38.264574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb11ffa4-de71-4bf6-982b-16802e27c2d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:38.265705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11ffa4-de71-4bf6-982b-16802e27c2d8 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:38.266223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11ffa4-de71-4bf6-982b-16802e27c2d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:38.266713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb11ffa4-de71-4bf6-982b-16802e27c2d8 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:38.267489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08487141-f545-4925-a010-274ab48e2508 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:38.267897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb11ffa4-de71-4bf6-982b-16802e27c2d8 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:38.270016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb11ffa4-de71-4bf6-982b-16802e27c2d8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:38.309920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1820bd-8f77-40d0-b6a1-f623e0d7c57e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:38.310820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1820bd-8f77-40d0-b6a1-f623e0d7c57e None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:38.311109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1820bd-8f77-40d0-b6a1-f623e0d7c57e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:38.311392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1820bd-8f77-40d0-b6a1-f623e0d7c57e None admin] PATH_INFO: `/v3/domains/default/users/05ecde7b5ffd440ea8bdbb9c0c6418c1/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:38.314360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1820bd-8f77-40d0-b6a1-f623e0d7c57e None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=05ecde7b5ffd440ea8bdbb9c0c6418c1, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:38.315532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1820bd-8f77-40d0-b6a1-f623e0d7c57e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:38.393793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 19/38] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:44:38 2022] POST /identity/v3/projects => generated 301 bytes in 295 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:44:38.393793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fccf8bb-7b13-45a8-8036-f6c026ecf231 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:38.393793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fccf8bb-7b13-45a8-8036-f6c026ecf231 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:38.393793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fccf8bb-7b13-45a8-8036-f6c026ecf231 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:38.393793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 20/39] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:38.393793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ce0520-2a78-4093-b039-9860d0a6eb0d None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:38.394312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ce0520-2a78-4093-b039-9860d0a6eb0d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:38.394389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ce0520-2a78-4093-b039-9860d0a6eb0d None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:38.395666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 21/40] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:38.399069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96c7158-2aa8-403d-ae7e-54476b90fe41 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:38.399817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96c7158-2aa8-403d-ae7e-54476b90fe41 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:38.400272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96c7158-2aa8-403d-ae7e-54476b90fe41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:38.418128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 20/41] 10.222.0.25 () {58 vars in 1361 bytes} [Fri Jun 10 12:44:38 2022] PUT /identity/v3/domains/default/users/05ecde7b5ffd440ea8bdbb9c0c6418c1/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:44:38.419828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5608a91-58b8-49dc-a0e0-5761af2c8a45 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:38.420128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5608a91-58b8-49dc-a0e0-5761af2c8a45 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:38.420412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5608a91-58b8-49dc-a0e0-5761af2c8a45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:38.423321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e96c7158-2aa8-403d-ae7e-54476b90fe41 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:38.425048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e96c7158-2aa8-403d-ae7e-54476b90fe41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:38.448262 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c5608a91-58b8-49dc-a0e0-5761af2c8a45 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:38.451225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c5608a91-58b8-49dc-a0e0-5761af2c8a45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:38.480027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:38.537994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 22/42] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:38.567632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 21/43] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:38.574132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96c7158-2aa8-403d-ae7e-54476b90fe41 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:38.592271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5608a91-58b8-49dc-a0e0-5761af2c8a45 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:38.681977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:38.695234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96c7158-2aa8-403d-ae7e-54476b90fe41 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:38.714694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5608a91-58b8-49dc-a0e0-5761af2c8a45 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:38.732985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb9049e-9cc6-4f6b-b560-e5f862b6bc4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:38.733670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb9049e-9cc6-4f6b-b560-e5f862b6bc4e None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:38.733961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb9049e-9cc6-4f6b-b560-e5f862b6bc4e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:38.734245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb9049e-9cc6-4f6b-b560-e5f862b6bc4e None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:38.735117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb9049e-9cc6-4f6b-b560-e5f862b6bc4e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:38.736524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb9049e-9cc6-4f6b-b560-e5f862b6bc4e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:38.748471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2e8d42-351c-4264-9bd5-eee7322a78de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:38.749373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2e8d42-351c-4264-9bd5-eee7322a78de None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:38.749684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2e8d42-351c-4264-9bd5-eee7322a78de None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:38.749964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2e8d42-351c-4264-9bd5-eee7322a78de None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:38.751162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2e8d42-351c-4264-9bd5-eee7322a78de None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:38.752625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2e8d42-351c-4264-9bd5-eee7322a78de None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:38.754278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 22/44] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:38 2022] GET /identity/v3/domains/default => generated 199 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:38.759979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2e8d42-351c-4264-9bd5-eee7322a78de None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:38.784939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:38.844718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 23/45] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Jun 10 12:44:38 2022] POST /identity/v3/roles => generated 223 bytes in 271 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:44:38.864912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2e8d42-351c-4264-9bd5-eee7322a78de None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:38.909297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20afc10b-f5ae-4233-a70a-e70f25706c57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:38.909987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20afc10b-f5ae-4233-a70a-e70f25706c57 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:38.910282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20afc10b-f5ae-4233-a70a-e70f25706c57 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:38.910566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20afc10b-f5ae-4233-a70a-e70f25706c57 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:38.911303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20afc10b-f5ae-4233-a70a-e70f25706c57 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:38.912897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20afc10b-f5ae-4233-a70a-e70f25706c57 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:39.029334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 23/46] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:44:38 2022] POST /identity/v3/projects => generated 312 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:44:40.570596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ca1a9-e6d3-4763-a31f-c72a817d323c None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:40.571094 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ca1a9-e6d3-4763-a31f-c72a817d323c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:40.571174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d95ca1a9-e6d3-4763-a31f-c72a817d323c None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:40.572544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 24/47] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:40.578287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a139af9-7b9d-458c-8032-6ff0959d9933 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:40.578687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a139af9-7b9d-458c-8032-6ff0959d9933 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:40.578866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a139af9-7b9d-458c-8032-6ff0959d9933 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:40.595920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-0a139af9-7b9d-458c-8032-6ff0959d9933 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:40.597094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0a139af9-7b9d-458c-8032-6ff0959d9933 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:40.661637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 24/48] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:40.684670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ca1a9-e6d3-4763-a31f-c72a817d323c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:40.783763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d95ca1a9-e6d3-4763-a31f-c72a817d323c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:40.816928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc1f33d-2b1f-4e0e-8d24-6a0827610db8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:40.817713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc1f33d-2b1f-4e0e-8d24-6a0827610db8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:40.817998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc1f33d-2b1f-4e0e-8d24-6a0827610db8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:40.818279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc1f33d-2b1f-4e0e-8d24-6a0827610db8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:40.828213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc1f33d-2b1f-4e0e-8d24-6a0827610db8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-0cc1f33d-2b1f-4e0e-8d24-6a0827610db8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:40.835783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:40.836756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:40.837636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 25/49] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:44:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:40.843506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a139af9-7b9d-458c-8032-6ff0959d9933 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:40.846626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:40.939699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a139af9-7b9d-458c-8032-6ff0959d9933 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:40.972119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c2fda2-e28b-4079-8c54-76a4bbd16687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:40.972809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c2fda2-e28b-4079-8c54-76a4bbd16687 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:40.973125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c2fda2-e28b-4079-8c54-76a4bbd16687 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:40.973405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c2fda2-e28b-4079-8c54-76a4bbd16687 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:40.974163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c2fda2-e28b-4079-8c54-76a4bbd16687 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:40.974472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c2fda2-e28b-4079-8c54-76a4bbd16687 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:40.975876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c2fda2-e28b-4079-8c54-76a4bbd16687 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:40.984831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 25/50] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:40.990205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc1f33d-2b1f-4e0e-8d24-6a0827610db8 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:41.001604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:41.089363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc1f33d-2b1f-4e0e-8d24-6a0827610db8 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:41.126293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb2752d-7f9f-4f1d-b76f-ad5c747efffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:41.127098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb2752d-7f9f-4f1d-b76f-ad5c747efffb None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:41.127384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb2752d-7f9f-4f1d-b76f-ad5c747efffb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:41.127663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb2752d-7f9f-4f1d-b76f-ad5c747efffb None admin] PATH_INFO: `/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:41.128884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb2752d-7f9f-4f1d-b76f-ad5c747efffb None admin] RBAC: Authorizing `identity:get_user(user_id=05ecde7b5ffd440ea8bdbb9c0c6418c1)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:41.130363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb2752d-7f9f-4f1d-b76f-ad5c747efffb None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:41.132001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 26/51] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:40 2022] GET /identity/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1 => generated 247 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:41.137512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c2fda2-e28b-4079-8c54-76a4bbd16687 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:41.158407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1bf289-29f4-4e33-9990-e37d4f8e2654 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:41.158697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1bf289-29f4-4e33-9990-e37d4f8e2654 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:41.159845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1bf289-29f4-4e33-9990-e37d4f8e2654 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:41.160547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 27/52] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:44:41.165309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bde4d4-b8d5-401b-9008-19a7ef502e4c None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:41.165600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bde4d4-b8d5-401b-9008-19a7ef502e4c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:41.165884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bde4d4-b8d5-401b-9008-19a7ef502e4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:41.183081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-18bde4d4-b8d5-401b-9008-19a7ef502e4c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:41.184846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-18bde4d4-b8d5-401b-9008-19a7ef502e4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:41.203354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:41.238582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c2fda2-e28b-4079-8c54-76a4bbd16687 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:41.252436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 28/53] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:44:41.270575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc919aa-a414-4700-8b2b-4caf46466ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:41.271364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc919aa-a414-4700-8b2b-4caf46466ea9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:41.271650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc919aa-a414-4700-8b2b-4caf46466ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:41.271930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc919aa-a414-4700-8b2b-4caf46466ea9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:41.273158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc919aa-a414-4700-8b2b-4caf46466ea9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:41.274694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc919aa-a414-4700-8b2b-4caf46466ea9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:41.275032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bde4d4-b8d5-401b-9008-19a7ef502e4c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:41.276301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 26/54] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:41.282145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc919aa-a414-4700-8b2b-4caf46466ea9 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:41.283013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:41.375171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bde4d4-b8d5-401b-9008-19a7ef502e4c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:41.380824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc919aa-a414-4700-8b2b-4caf46466ea9 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:41.409812 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e647d2-f88a-4775-911d-6a41228a45fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:41.410601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e647d2-f88a-4775-911d-6a41228a45fb None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:41.410893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e647d2-f88a-4775-911d-6a41228a45fb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:41.411174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e647d2-f88a-4775-911d-6a41228a45fb None admin] PATH_INFO: `/v3/domains/default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:41.412352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e647d2-f88a-4775-911d-6a41228a45fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:41.413848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e647d2-f88a-4775-911d-6a41228a45fb None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:41.414581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-946f4bd2-d14a-46d5-85dd-c8dd4fe2a252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:41.415232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f4bd2-d14a-46d5-85dd-c8dd4fe2a252 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:41.415469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 29/55] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:41.415587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f4bd2-d14a-46d5-85dd-c8dd4fe2a252 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:41.415867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-946f4bd2-d14a-46d5-85dd-c8dd4fe2a252 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:41.416791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946f4bd2-d14a-46d5-85dd-c8dd4fe2a252 None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.domain.id=default, user.id=05ecde7b5ffd440ea8bdbb9c0c6418c1) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:41.417119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946f4bd2-d14a-46d5-85dd-c8dd4fe2a252 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:41.419083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-946f4bd2-d14a-46d5-85dd-c8dd4fe2a252 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:41.421438 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e647d2-f88a-4775-911d-6a41228a45fb None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:41.421605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:41.428031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 27/56] 10.222.0.25 () {56 vars in 1322 bytes} [Fri Jun 10 12:44:41 2022] GET /identity/v3/role_assignments?user.id=05ecde7b5ffd440ea8bdbb9c0c6418c1&scope.domain.id=default&role.id=766799d1b6c049168590571f1a959b3b => generated 530 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:41.503447 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:41.529084 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e647d2-f88a-4775-911d-6a41228a45fb None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:41.564730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a16b97a-a373-40a9-b880-fc061ecf3ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:41.565435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a16b97a-a373-40a9-b880-fc061ecf3ff0 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:41.565724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a16b97a-a373-40a9-b880-fc061ecf3ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:41.566006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a16b97a-a373-40a9-b880-fc061ecf3ff0 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:41.566734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a16b97a-a373-40a9-b880-fc061ecf3ff0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:41.568208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a16b97a-a373-40a9-b880-fc061ecf3ff0 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:41.649457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 30/57] 10.222.0.25 () {60 vars in 1140 bytes} [Fri Jun 10 12:44:41 2022] POST /identity/v3/projects => generated 298 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:44:43.743334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da59ec54-fdfb-4ca4-b014-aafb5dd31fc5 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:43.743876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da59ec54-fdfb-4ca4-b014-aafb5dd31fc5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:43.743876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da59ec54-fdfb-4ca4-b014-aafb5dd31fc5 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:43.745314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 28/58] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:43.750787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d0497a-6893-4ede-a299-a2a73af65142 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:43.751183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d0497a-6893-4ede-a299-a2a73af65142 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:43.751367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d0497a-6893-4ede-a299-a2a73af65142 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:43.768325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-c8d0497a-6893-4ede-a299-a2a73af65142 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:43.769560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c8d0497a-6893-4ede-a299-a2a73af65142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:43.847877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 31/59] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:43.872044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da59ec54-fdfb-4ca4-b014-aafb5dd31fc5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:43.970302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da59ec54-fdfb-4ca4-b014-aafb5dd31fc5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:44.002659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-403ef742-7718-4614-b8cf-d7d5bc780ab0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:44.003392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ef742-7718-4614-b8cf-d7d5bc780ab0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:44.003679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ef742-7718-4614-b8cf-d7d5bc780ab0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:44.003960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-403ef742-7718-4614-b8cf-d7d5bc780ab0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:44.005159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403ef742-7718-4614-b8cf-d7d5bc780ab0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:44.006618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-403ef742-7718-4614-b8cf-d7d5bc780ab0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:44.008206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 29/60] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:44.013615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d0497a-6893-4ede-a299-a2a73af65142 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:44.067881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:44.109752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d0497a-6893-4ede-a299-a2a73af65142 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:44.142364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e1d533-94af-47d5-afe4-a75d1483cec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:44.143027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e1d533-94af-47d5-afe4-a75d1483cec3 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:44.143318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e1d533-94af-47d5-afe4-a75d1483cec3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:44.143604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e1d533-94af-47d5-afe4-a75d1483cec3 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:44.144310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e1d533-94af-47d5-afe4-a75d1483cec3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:44.145836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e1d533-94af-47d5-afe4-a75d1483cec3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:44.313114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 32/61] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:44:44 2022] POST /identity/v3/users => generated 275 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:44:47.484974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7004897-a255-45f1-b192-34975da61633 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:47.485465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7004897-a255-45f1-b192-34975da61633 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:47.485538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7004897-a255-45f1-b192-34975da61633 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:47.487054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 30/62] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:47.505601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a336639-4a0b-4166-b102-596d7a2e8f0e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:47.506188 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a336639-4a0b-4166-b102-596d7a2e8f0e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:47.506188 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a336639-4a0b-4166-b102-596d7a2e8f0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:47.525056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-2a336639-4a0b-4166-b102-596d7a2e8f0e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:47.526269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2a336639-4a0b-4166-b102-596d7a2e8f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:47.611433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 33/63] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:47.654954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7004897-a255-45f1-b192-34975da61633 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:47.753084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:47.809900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7004897-a255-45f1-b192-34975da61633 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:47.859107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75b4ed5-ee2d-45da-80f0-5c3fef1d9eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:47.859932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75b4ed5-ee2d-45da-80f0-5c3fef1d9eda None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:47.860223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75b4ed5-ee2d-45da-80f0-5c3fef1d9eda None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:47.860515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75b4ed5-ee2d-45da-80f0-5c3fef1d9eda None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:47.878972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75b4ed5-ee2d-45da-80f0-5c3fef1d9eda None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-d75b4ed5-ee2d-45da-80f0-5c3fef1d9eda None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:47.888503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:47.889386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:47.890135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:44:47.893866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 31/64] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:47 2022] GET /identity/v3/roles/member => generated 84 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:44:47.899869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a336639-4a0b-4166-b102-596d7a2e8f0e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:47.908666 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b8569d-7ecc-4e3a-a863-053ac1b1f45d None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:47.909006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b8569d-7ecc-4e3a-a863-053ac1b1f45d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:47.909297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b8569d-7ecc-4e3a-a863-053ac1b1f45d None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:47.910678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 32/65] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:44:47.913792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5110f5ea-3df8-429b-b31d-cc771d61f82f None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:47.914449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5110f5ea-3df8-429b-b31d-cc771d61f82f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:47.914449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5110f5ea-3df8-429b-b31d-cc771d61f82f None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:47.916037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 33/66] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:44:47.917455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3298c82-79e8-4a1b-b387-5d63df86f28d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:47.917760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3298c82-79e8-4a1b-b387-5d63df86f28d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:47.918568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3298c82-79e8-4a1b-b387-5d63df86f28d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:47.936302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-b3298c82-79e8-4a1b-b387-5d63df86f28d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:47.937549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b3298c82-79e8-4a1b-b387-5d63df86f28d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:47.961642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:48.022559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 34/67] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:48.030599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c46105c-fc4c-4fed-bcf7-9e5fad5b4358 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.030936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c46105c-fc4c-4fed-bcf7-9e5fad5b4358 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.032596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c46105c-fc4c-4fed-bcf7-9e5fad5b4358 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.053552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a336639-4a0b-4166-b102-596d7a2e8f0e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.059561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-7c46105c-fc4c-4fed-bcf7-9e5fad5b4358 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:48.061602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7c46105c-fc4c-4fed-bcf7-9e5fad5b4358 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:48.096329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd5f766-a9ed-4d0e-85d0-063f425937db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:48.097266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd5f766-a9ed-4d0e-85d0-063f425937db None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.097701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd5f766-a9ed-4d0e-85d0-063f425937db None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.097982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd5f766-a9ed-4d0e-85d0-063f425937db None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.098734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd5f766-a9ed-4d0e-85d0-063f425937db None admin] RBAC: Adding query filter params (name=member) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:48.099045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd5f766-a9ed-4d0e-85d0-063f425937db None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:48.100446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd5f766-a9ed-4d0e-85d0-063f425937db None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:48.108551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 34/68] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:44:47 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:48.109470 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd5f766-a9ed-4d0e-85d0-063f425937db None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:48.118981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:48.141478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 35/69] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:48.141478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35b096f-1744-492d-b6de-3c30def1eb72 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.141478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35b096f-1744-492d-b6de-3c30def1eb72 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.141478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35b096f-1744-492d-b6de-3c30def1eb72 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.141478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 36/70] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:48.141478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc2211b-21cc-4091-813b-55a01aaf55ee None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.141478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc2211b-21cc-4091-813b-55a01aaf55ee None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.141942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc2211b-21cc-4091-813b-55a01aaf55ee None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.141942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 37/71] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:48 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:48.143505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc2211b-21cc-4091-813b-55a01aaf55ee None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:48.215847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd5f766-a9ed-4d0e-85d0-063f425937db None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.218152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:48.250607 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17bb756-8358-49d7-b7f7-7e80fc106cc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:48.251377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17bb756-8358-49d7-b7f7-7e80fc106cc5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.251662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17bb756-8358-49d7-b7f7-7e80fc106cc5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.251942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17bb756-8358-49d7-b7f7-7e80fc106cc5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.255489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc2211b-21cc-4091-813b-55a01aaf55ee None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.262445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17bb756-8358-49d7-b7f7-7e80fc106cc5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-f17bb756-8358-49d7-b7f7-7e80fc106cc5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:48.270248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:48.271078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:48.271847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 35/72] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:44:48 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:48.273920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49537a88-1283-4dc7-93ed-14bbf9523ff7 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.274216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49537a88-1283-4dc7-93ed-14bbf9523ff7 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.274504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49537a88-1283-4dc7-93ed-14bbf9523ff7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.289796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22560dd2-d0de-4de8-86ef-a09fb55a5834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:48.290665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22560dd2-d0de-4de8-86ef-a09fb55a5834 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.290954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22560dd2-d0de-4de8-86ef-a09fb55a5834 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.291141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-49537a88-1283-4dc7-93ed-14bbf9523ff7 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:48.291229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22560dd2-d0de-4de8-86ef-a09fb55a5834 None admin] PATH_INFO: `/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.292356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-49537a88-1283-4dc7-93ed-14bbf9523ff7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:48.305765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22560dd2-d0de-4de8-86ef-a09fb55a5834 None admin] RBAC: Authorizing `identity:get_user(user_id=5c8b0f52ee8b4d4a82703c331b1da20e)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:48.307289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22560dd2-d0de-4de8-86ef-a09fb55a5834 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:48.309493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 38/73] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:48 2022] GET /identity/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e => generated 275 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:48.310684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0d3700-87c6-45f7-ae03-946fbb799ab6 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.310976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0d3700-87c6-45f7-ae03-946fbb799ab6 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.311258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0d3700-87c6-45f7-ae03-946fbb799ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.328675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-af0d3700-87c6-45f7-ae03-946fbb799ab6 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:48.329893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-af0d3700-87c6-45f7-ae03-946fbb799ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:48.358924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 36/74] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:48.359770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49537a88-1283-4dc7-93ed-14bbf9523ff7 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:48.374668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:48.401374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 39/75] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:48.402046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0d3700-87c6-45f7-ae03-946fbb799ab6 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:48.461947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:48.468492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49537a88-1283-4dc7-93ed-14bbf9523ff7 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.501905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f492ad0-6df9-435a-b02a-ad11a294304d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:48.502955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f492ad0-6df9-435a-b02a-ad11a294304d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.503236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f492ad0-6df9-435a-b02a-ad11a294304d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.503512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f492ad0-6df9-435a-b02a-ad11a294304d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.513833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f492ad0-6df9-435a-b02a-ad11a294304d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:48.517133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0d3700-87c6-45f7-ae03-946fbb799ab6 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-9f492ad0-6df9-435a-b02a-ad11a294304d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:48.521859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:48.522886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 37/76] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:44:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:48.523819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f492ad0-6df9-435a-b02a-ad11a294304d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:48.552192 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3529b3-a169-446d-8b19-ac01c11e06a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:48.552895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3529b3-a169-446d-8b19-ac01c11e06a9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.553196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3529b3-a169-446d-8b19-ac01c11e06a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.553465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc3529b3-a169-446d-8b19-ac01c11e06a9 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.554220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3529b3-a169-446d-8b19-ac01c11e06a9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:48.554535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3529b3-a169-446d-8b19-ac01c11e06a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:48.555944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc3529b3-a169-446d-8b19-ac01c11e06a9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:48.564422 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 40/77] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:48.565159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3529b3-a169-446d-8b19-ac01c11e06a9 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:48.599534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:48.620142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f492ad0-6df9-435a-b02a-ad11a294304d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.653117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb7163a-abac-4374-8c77-22053456a7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:48.653694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb7163a-abac-4374-8c77-22053456a7a4 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.653980 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb7163a-abac-4374-8c77-22053456a7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.654262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb7163a-abac-4374-8c77-22053456a7a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.663783 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc3529b3-a169-446d-8b19-ac01c11e06a9 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.678232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:48.697403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26926b9-67cb-40dc-9fbf-86a403e4db1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:48.698191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26926b9-67cb-40dc-9fbf-86a403e4db1b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.708091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26926b9-67cb-40dc-9fbf-86a403e4db1b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.708091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26926b9-67cb-40dc-9fbf-86a403e4db1b None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.708091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26926b9-67cb-40dc-9fbf-86a403e4db1b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:48.708091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26926b9-67cb-40dc-9fbf-86a403e4db1b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:48.708091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 41/78] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:48.708091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26926b9-67cb-40dc-9fbf-86a403e4db1b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:48.720595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb7163a-abac-4374-8c77-22053456a7a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:48.722181 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb7163a-abac-4374-8c77-22053456a7a4 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:48.809993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26926b9-67cb-40dc-9fbf-86a403e4db1b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.812178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:48.825863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 38/79] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:44:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:44:48.826892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb7163a-abac-4374-8c77-22053456a7a4 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:48.846962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d003a8c-fe56-4e9e-83f4-af3aaacf9d92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:48.847750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d003a8c-fe56-4e9e-83f4-af3aaacf9d92 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.848035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d003a8c-fe56-4e9e-83f4-af3aaacf9d92 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.848313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d003a8c-fe56-4e9e-83f4-af3aaacf9d92 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.857991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d003a8c-fe56-4e9e-83f4-af3aaacf9d92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-3d003a8c-fe56-4e9e-83f4-af3aaacf9d92 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:48.865541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:48.866539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 42/80] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:48.867552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d003a8c-fe56-4e9e-83f4-af3aaacf9d92 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:48.883718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:48.935690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb7163a-abac-4374-8c77-22053456a7a4 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.968464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d003a8c-fe56-4e9e-83f4-af3aaacf9d92 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:48.968924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c547eb7-ced0-4010-90cb-088f839ebd50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:48.969590 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c547eb7-ced0-4010-90cb-088f839ebd50 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:48.969884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c547eb7-ced0-4010-90cb-088f839ebd50 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:48.970165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c547eb7-ced0-4010-90cb-088f839ebd50 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:48.970914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c547eb7-ced0-4010-90cb-088f839ebd50 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:48.971227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c547eb7-ced0-4010-90cb-088f839ebd50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:48.972639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c547eb7-ced0-4010-90cb-088f839ebd50 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:48.980415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 39/81] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:44:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:48.981850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c547eb7-ced0-4010-90cb-088f839ebd50 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.004583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f68a902-4da0-42a8-abc2-14b171c10e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.005480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f68a902-4da0-42a8-abc2-14b171c10e47 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.005767 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f68a902-4da0-42a8-abc2-14b171c10e47 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.006048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f68a902-4da0-42a8-abc2-14b171c10e47 None admin] PATH_INFO: `/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.007242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f68a902-4da0-42a8-abc2-14b171c10e47 None admin] RBAC: Authorizing `identity:get_user(user_id=05ecde7b5ffd440ea8bdbb9c0c6418c1)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:49.008714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f68a902-4da0-42a8-abc2-14b171c10e47 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:49.010361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 43/82] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:48 2022] GET /identity/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1 => generated 247 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:49.013638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f68a902-4da0-42a8-abc2-14b171c10e47 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.047366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:49.093052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c547eb7-ced0-4010-90cb-088f839ebd50 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:49.109604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:49.114637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f68a902-4da0-42a8-abc2-14b171c10e47 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:49.126624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ab098b-0e38-431a-b4cf-f8e9e883a8f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.127367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab098b-0e38-431a-b4cf-f8e9e883a8f7 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.127650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab098b-0e38-431a-b4cf-f8e9e883a8f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.127935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ab098b-0e38-431a-b4cf-f8e9e883a8f7 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.128658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ab098b-0e38-431a-b4cf-f8e9e883a8f7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:49.130218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ab098b-0e38-431a-b4cf-f8e9e883a8f7 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:49.147575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9399dc02-374e-42a2-9d9e-e4fefa1026e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.148425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9399dc02-374e-42a2-9d9e-e4fefa1026e0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.148708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9399dc02-374e-42a2-9d9e-e4fefa1026e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.149087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9399dc02-374e-42a2-9d9e-e4fefa1026e0 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.150355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9399dc02-374e-42a2-9d9e-e4fefa1026e0 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:49.151840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9399dc02-374e-42a2-9d9e-e4fefa1026e0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:49.154063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 44/83] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:49 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:49.155064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9399dc02-374e-42a2-9d9e-e4fefa1026e0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.214670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 40/84] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:44:48 2022] POST /identity/v3/users => generated 289 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:44:49.233891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ab098b-0e38-431a-b4cf-f8e9e883a8f7 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.251747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:49.254914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9399dc02-374e-42a2-9d9e-e4fefa1026e0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:49.296586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5f1d1f-1b2f-4139-9be6-385673228f57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.297462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5f1d1f-1b2f-4139-9be6-385673228f57 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.297748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5f1d1f-1b2f-4139-9be6-385673228f57 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.298028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5f1d1f-1b2f-4139-9be6-385673228f57 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.298795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5f1d1f-1b2f-4139-9be6-385673228f57 None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:49.299108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5f1d1f-1b2f-4139-9be6-385673228f57 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:49.300562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5f1d1f-1b2f-4139-9be6-385673228f57 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:49.314424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 45/85] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:44:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:49.314424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5f1d1f-1b2f-4139-9be6-385673228f57 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.357062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ab098b-0e38-431a-b4cf-f8e9e883a8f7 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:49.404156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2526f82-ae06-4fee-919a-e2f45548e3f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.405774 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2526f82-ae06-4fee-919a-e2f45548e3f8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.406066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2526f82-ae06-4fee-919a-e2f45548e3f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.406820 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2526f82-ae06-4fee-919a-e2f45548e3f8 None admin] PATH_INFO: `/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.409220 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2526f82-ae06-4fee-919a-e2f45548e3f8 None admin] RBAC: Authorizing `identity:get_user(user_id=5c8b0f52ee8b4d4a82703c331b1da20e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:49.410350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2526f82-ae06-4fee-919a-e2f45548e3f8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:49.412200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 41/86] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:49 2022] GET /identity/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:49.413012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2526f82-ae06-4fee-919a-e2f45548e3f8 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.438182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5f1d1f-1b2f-4139-9be6-385673228f57 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:49.481946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:49.495217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585e471c-21d0-451c-af5b-b895dbc0b9f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.495876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e471c-21d0-451c-af5b-b895dbc0b9f5 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.496197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e471c-21d0-451c-af5b-b895dbc0b9f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.497263 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585e471c-21d0-451c-af5b-b895dbc0b9f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.533533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:49.552368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2526f82-ae06-4fee-919a-e2f45548e3f8 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:49.583533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585e471c-21d0-451c-af5b-b895dbc0b9f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:49.585392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585e471c-21d0-451c-af5b-b895dbc0b9f5 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:49.593400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb91e4b-2b76-452e-88ae-08193af84ebe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.594320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb91e4b-2b76-452e-88ae-08193af84ebe None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.594717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb91e4b-2b76-452e-88ae-08193af84ebe None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.595116 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb91e4b-2b76-452e-88ae-08193af84ebe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.596089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb91e4b-2b76-452e-88ae-08193af84ebe None admin] RBAC: Adding query filter params (role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569, user.id=5c8b0f52ee8b4d4a82703c331b1da20e) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:49.596622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb91e4b-2b76-452e-88ae-08193af84ebe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:49.598360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb91e4b-2b76-452e-88ae-08193af84ebe None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:49.604971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 42/87] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:44:49 2022] GET /identity/v3/role_assignments?user.id=5c8b0f52ee8b4d4a82703c331b1da20e&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 259 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:49.606638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb91e4b-2b76-452e-88ae-08193af84ebe None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.722256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb91e4b-2b76-452e-88ae-08193af84ebe None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:49.725937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 46/88] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:44:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:44:49.727629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:49.734402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585e471c-21d0-451c-af5b-b895dbc0b9f5 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.765329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff04b347-3db2-4165-84a1-a1b57ceba603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.766407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04b347-3db2-4165-84a1-a1b57ceba603 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.766758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04b347-3db2-4165-84a1-a1b57ceba603 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.767115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff04b347-3db2-4165-84a1-a1b57ceba603 None admin] PATH_INFO: `/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.768524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff04b347-3db2-4165-84a1-a1b57ceba603 None admin] RBAC: Authorizing `identity:get_user(user_id=5c8b0f52ee8b4d4a82703c331b1da20e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:49.770148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff04b347-3db2-4165-84a1-a1b57ceba603 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:49.772134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 43/89] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:49 2022] GET /identity/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:49.773110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff04b347-3db2-4165-84a1-a1b57ceba603 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.809563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:49.852269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585e471c-21d0-451c-af5b-b895dbc0b9f5 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:49.889514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e9fff3-bbec-48bc-baa4-089b6648822e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.890536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e9fff3-bbec-48bc-baa4-089b6648822e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.890756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff04b347-3db2-4165-84a1-a1b57ceba603 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:49.891246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e9fff3-bbec-48bc-baa4-089b6648822e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.891711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e9fff3-bbec-48bc-baa4-089b6648822e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.951090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df7dbf9-aaf0-49ab-a145-0e0027fdc422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:49.951966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df7dbf9-aaf0-49ab-a145-0e0027fdc422 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:49.956208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df7dbf9-aaf0-49ab-a145-0e0027fdc422 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:49.956208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df7dbf9-aaf0-49ab-a145-0e0027fdc422 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:49.956208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df7dbf9-aaf0-49ab-a145-0e0027fdc422 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:49.956208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df7dbf9-aaf0-49ab-a145-0e0027fdc422 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:49.957465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 44/90] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:49 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:49.958442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df7dbf9-aaf0-49ab-a145-0e0027fdc422 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:49.995155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e9fff3-bbec-48bc-baa4-089b6648822e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:50.001243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53e9fff3-bbec-48bc-baa4-089b6648822e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:50.038797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:50.141501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:50.153717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df7dbf9-aaf0-49ab-a145-0e0027fdc422 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:50.224095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 47/91] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:44:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:44:50.227914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e9fff3-bbec-48bc-baa4-089b6648822e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:50.229280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ec214f-fc6f-41d9-9a34-939afa0eb73d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:50.230371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ec214f-fc6f-41d9-9a34-939afa0eb73d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:50.233193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ec214f-fc6f-41d9-9a34-939afa0eb73d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:50.233714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ec214f-fc6f-41d9-9a34-939afa0eb73d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:50.330303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ec214f-fc6f-41d9-9a34-939afa0eb73d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:50.332115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ec214f-fc6f-41d9-9a34-939afa0eb73d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:50.380261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:50.431562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e9fff3-bbec-48bc-baa4-089b6648822e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:50.445513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 45/92] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:44:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:44:50.446613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ec214f-fc6f-41d9-9a34-939afa0eb73d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:50.509763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae46edd-88ea-40b9-ac1f-939636d08518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:50.510774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae46edd-88ea-40b9-ac1f-939636d08518 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:50.511253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae46edd-88ea-40b9-ac1f-939636d08518 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:50.511991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae46edd-88ea-40b9-ac1f-939636d08518 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:50.513174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae46edd-88ea-40b9-ac1f-939636d08518 None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569, user.id=05ecde7b5ffd440ea8bdbb9c0c6418c1) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:50.513683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae46edd-88ea-40b9-ac1f-939636d08518 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:50.515433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae46edd-88ea-40b9-ac1f-939636d08518 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:50.533681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 48/93] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:44:50 2022] GET /identity/v3/role_assignments?user.id=05ecde7b5ffd440ea8bdbb9c0c6418c1&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=766799d1b6c049168590571f1a959b3b => generated 259 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:50.534520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae46edd-88ea-40b9-ac1f-939636d08518 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:50.608596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:50.623445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ec214f-fc6f-41d9-9a34-939afa0eb73d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:50.737595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f56a29-0a7d-4395-aef2-50316ed3770d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:50.738493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f56a29-0a7d-4395-aef2-50316ed3770d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:50.738787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f56a29-0a7d-4395-aef2-50316ed3770d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:50.739075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f56a29-0a7d-4395-aef2-50316ed3770d None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:50.740386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f56a29-0a7d-4395-aef2-50316ed3770d None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:50.744807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f56a29-0a7d-4395-aef2-50316ed3770d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:50.744807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 46/94] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:50 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:50.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f56a29-0a7d-4395-aef2-50316ed3770d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:50.760510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae46edd-88ea-40b9-ac1f-939636d08518 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:50.769976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:50.819374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a1e52b-1be8-4f4d-8fa2-c0599292fd75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:50.820244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a1e52b-1be8-4f4d-8fa2-c0599292fd75 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:50.820529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a1e52b-1be8-4f4d-8fa2-c0599292fd75 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:50.820832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a1e52b-1be8-4f4d-8fa2-c0599292fd75 None admin] PATH_INFO: `/v3/projects/edf1e3eb7289421a9782aa94730b2021` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:50.822665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a1e52b-1be8-4f4d-8fa2-c0599292fd75 None admin] RBAC: Authorizing `identity:get_project(project_id=edf1e3eb7289421a9782aa94730b2021)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:50.825735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a1e52b-1be8-4f4d-8fa2-c0599292fd75 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:50.827527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 49/95] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:50 2022] GET /identity/v3/projects/edf1e3eb7289421a9782aa94730b2021 => generated 312 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:50.838495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a1e52b-1be8-4f4d-8fa2-c0599292fd75 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:50.963417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f56a29-0a7d-4395-aef2-50316ed3770d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:51.025122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:51.028123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c885648-71e6-4754-8536-17c612ba01c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:51.028965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c885648-71e6-4754-8536-17c612ba01c3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:51.029141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c885648-71e6-4754-8536-17c612ba01c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:51.029380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c885648-71e6-4754-8536-17c612ba01c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:51.030480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c885648-71e6-4754-8536-17c612ba01c3 None admin] RBAC: Adding query filter params (role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569, user.id=5c8b0f52ee8b4d4a82703c331b1da20e) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:51.030797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c885648-71e6-4754-8536-17c612ba01c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:51.034234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c885648-71e6-4754-8536-17c612ba01c3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:51.036277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a1e52b-1be8-4f4d-8fa2-c0599292fd75 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:51.047256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 47/96] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:44:50 2022] GET /identity/v3/role_assignments?user.id=5c8b0f52ee8b4d4a82703c331b1da20e&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 259 bytes in 297 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:51.087466 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfb72ae-f612-4625-aab3-eaeadd8d25f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:51.088117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfb72ae-f612-4625-aab3-eaeadd8d25f6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:51.088414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfb72ae-f612-4625-aab3-eaeadd8d25f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:51.088756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfb72ae-f612-4625-aab3-eaeadd8d25f6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:51.089640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfb72ae-f612-4625-aab3-eaeadd8d25f6 None admin] RBAC: Adding query filter params (role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=edf1e3eb7289421a9782aa94730b2021, user.id=5c8b0f52ee8b4d4a82703c331b1da20e) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:51.090075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfb72ae-f612-4625-aab3-eaeadd8d25f6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:51.094547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfb72ae-f612-4625-aab3-eaeadd8d25f6 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:51.101854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 50/97] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:44:50 2022] GET /identity/v3/role_assignments?user.id=5c8b0f52ee8b4d4a82703c331b1da20e&scope.project.id=edf1e3eb7289421a9782aa94730b2021&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 259 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:52.122028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef7a45a-5fc7-44c4-84e1-2e8a7c128012 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.122525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef7a45a-5fc7-44c4-84e1-2e8a7c128012 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.122596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef7a45a-5fc7-44c4-84e1-2e8a7c128012 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:52.123973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 48/98] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:52.134507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67a01d-a880-4687-bf92-fa7de3fc1ea5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.134969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67a01d-a880-4687-bf92-fa7de3fc1ea5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.135149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e67a01d-a880-4687-bf92-fa7de3fc1ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:52.145578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e5b606-0928-456c-b1d6-aada0034c853 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.145878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e5b606-0928-456c-b1d6-aada0034c853 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.146163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e5b606-0928-456c-b1d6-aada0034c853 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:52.147669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 49/99] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:44:52.154997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c42642f-d23e-48d1-97f9-f6265144df46 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.155288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c42642f-d23e-48d1-97f9-f6265144df46 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.155575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c42642f-d23e-48d1-97f9-f6265144df46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:52.170477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5e67a01d-a880-4687-bf92-fa7de3fc1ea5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:52.171721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5e67a01d-a880-4687-bf92-fa7de3fc1ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:52.181145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3c42642f-d23e-48d1-97f9-f6265144df46 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:52.182383 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3c42642f-d23e-48d1-97f9-f6265144df46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:52.299345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 51/100] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:52.310617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 50/101] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:44:52.332797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67a01d-a880-4687-bf92-fa7de3fc1ea5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:52.357512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c42642f-d23e-48d1-97f9-f6265144df46 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:52.359689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:52.489144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:52.521158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e67a01d-a880-4687-bf92-fa7de3fc1ea5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:52.586392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f95acf-7b17-4a1f-bd54-286fb00d9071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:52.587658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f95acf-7b17-4a1f-bd54-286fb00d9071 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.588139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f95acf-7b17-4a1f-bd54-286fb00d9071 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.588630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f95acf-7b17-4a1f-bd54-286fb00d9071 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:52.592670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f95acf-7b17-4a1f-bd54-286fb00d9071 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:52.594518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f95acf-7b17-4a1f-bd54-286fb00d9071 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:52.596391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 52/102] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:52.606583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f95acf-7b17-4a1f-bd54-286fb00d9071 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:52.607527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c42642f-d23e-48d1-97f9-f6265144df46 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:52.689658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c0cd83-03b2-44bc-9d7f-4311338deb81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:52.690523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c0cd83-03b2-44bc-9d7f-4311338deb81 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.690814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c0cd83-03b2-44bc-9d7f-4311338deb81 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.691097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c0cd83-03b2-44bc-9d7f-4311338deb81 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:52.710833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c0cd83-03b2-44bc-9d7f-4311338deb81 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-28c0cd83-03b2-44bc-9d7f-4311338deb81 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:52.719586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:52.720521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:52.721309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 51/103] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 364 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:44:52.729016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c0cd83-03b2-44bc-9d7f-4311338deb81 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:52.800692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:52.827186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f95acf-7b17-4a1f-bd54-286fb00d9071 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:52.839603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c0cd83-03b2-44bc-9d7f-4311338deb81 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:52.876636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab46667-7c3c-4fd9-8fac-fdfe339faafb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:52.877335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab46667-7c3c-4fd9-8fac-fdfe339faafb None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.877626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab46667-7c3c-4fd9-8fac-fdfe339faafb None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.877911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab46667-7c3c-4fd9-8fac-fdfe339faafb None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:52.878640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab46667-7c3c-4fd9-8fac-fdfe339faafb None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:52.886134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab46667-7c3c-4fd9-8fac-fdfe339faafb None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:52.902030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792f2443-dbd7-4c6d-97ed-9f193e2dfa6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:52.908803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792f2443-dbd7-4c6d-97ed-9f193e2dfa6d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.908803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792f2443-dbd7-4c6d-97ed-9f193e2dfa6d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.908803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792f2443-dbd7-4c6d-97ed-9f193e2dfa6d None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:52.908803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792f2443-dbd7-4c6d-97ed-9f193e2dfa6d None admin] RBAC: Adding query filter params (name=reader) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:52.908803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792f2443-dbd7-4c6d-97ed-9f193e2dfa6d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:52.908803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792f2443-dbd7-4c6d-97ed-9f193e2dfa6d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:52.915842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 52/104] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:44:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:52.916668 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba45534-d588-48df-bf93-fed4faa08d55 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.916994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba45534-d588-48df-bf93-fed4faa08d55 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.917283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba45534-d588-48df-bf93-fed4faa08d55 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:52.918838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 53/105] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:52.923517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba45534-d588-48df-bf93-fed4faa08d55 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:52.979554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 53/106] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:44:52 2022] POST /identity/v3/projects => generated 302 bytes in 374 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:44:52.981518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9780312-6491-446c-bd0b-eb1c1c7dff88 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:52.982041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9780312-6491-446c-bd0b-eb1c1c7dff88 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:52.982514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9780312-6491-446c-bd0b-eb1c1c7dff88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:53.042711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e9780312-6491-446c-bd0b-eb1c1c7dff88 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:53.044056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e9780312-6491-446c-bd0b-eb1c1c7dff88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:53.047606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:53.074722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba45534-d588-48df-bf93-fed4faa08d55 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:53.110900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:53.125599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86adb925-1325-4c5f-92b5-25f9cda74639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:53.126514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86adb925-1325-4c5f-92b5-25f9cda74639 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:53.126809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86adb925-1325-4c5f-92b5-25f9cda74639 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:53.127095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86adb925-1325-4c5f-92b5-25f9cda74639 None admin] PATH_INFO: `/v3/users/aa2f0b25d6414893bb607fdcecb88286` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:53.150693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86adb925-1325-4c5f-92b5-25f9cda74639 None admin] RBAC: Authorizing `identity:get_user(user_id=aa2f0b25d6414893bb607fdcecb88286)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:53.152264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86adb925-1325-4c5f-92b5-25f9cda74639 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:53.154255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 54/107] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:52 2022] GET /identity/v3/users/aa2f0b25d6414893bb607fdcecb88286 => generated 289 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:53.160081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86adb925-1325-4c5f-92b5-25f9cda74639 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:53.214047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 54/108] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:53.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9780312-6491-446c-bd0b-eb1c1c7dff88 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:53.283208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86adb925-1325-4c5f-92b5-25f9cda74639 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:53.346058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb66deae-8db9-49b9-be6b-ac74a2d2604f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:53.346733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb66deae-8db9-49b9-be6b-ac74a2d2604f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:53.347024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb66deae-8db9-49b9-be6b-ac74a2d2604f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:53.347311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb66deae-8db9-49b9-be6b-ac74a2d2604f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:53.392500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9780312-6491-446c-bd0b-eb1c1c7dff88 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:53.429985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff5d00e-2a3e-4de0-91d1-4f2ed2bfef57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:53.430792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff5d00e-2a3e-4de0-91d1-4f2ed2bfef57 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:53.431095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff5d00e-2a3e-4de0-91d1-4f2ed2bfef57 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:53.431285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:53.431402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff5d00e-2a3e-4de0-91d1-4f2ed2bfef57 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:53.438705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:53.443859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb66deae-8db9-49b9-be6b-ac74a2d2604f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:53.452992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff5d00e-2a3e-4de0-91d1-4f2ed2bfef57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:53.453111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb66deae-8db9-49b9-be6b-ac74a2d2604f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-bff5d00e-2a3e-4de0-91d1-4f2ed2bfef57 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:53.453579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:53.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:53.455544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:44:53.457584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 55/109] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:53.463308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff5d00e-2a3e-4de0-91d1-4f2ed2bfef57 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:53.650007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 55/110] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:44:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jun 10 12:44:53.656787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d3364b-839b-4839-84db-a0709419b6e6 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:53.658381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d3364b-839b-4839-84db-a0709419b6e6 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:53.659567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d3364b-839b-4839-84db-a0709419b6e6 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:53.661105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 56/111] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:53 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:53.666795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd168d-138a-4d56-b063-3b1185dfff08 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:53.667252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd168d-138a-4d56-b063-3b1185dfff08 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:53.667678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbd168d-138a-4d56-b063-3b1185dfff08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:53.678046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff5d00e-2a3e-4de0-91d1-4f2ed2bfef57 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:53.686332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3fbd168d-138a-4d56-b063-3b1185dfff08 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:53.687701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3fbd168d-138a-4d56-b063-3b1185dfff08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:53.764233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 57/112] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:53.765354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbd168d-138a-4d56-b063-3b1185dfff08 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:53.766205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ff1017-30f4-4dd1-9a2e-46db9d70e57f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:53.766887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ff1017-30f4-4dd1-9a2e-46db9d70e57f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:53.767176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ff1017-30f4-4dd1-9a2e-46db9d70e57f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:53.767459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ff1017-30f4-4dd1-9a2e-46db9d70e57f None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:53.768506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ff1017-30f4-4dd1-9a2e-46db9d70e57f None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:53.768848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ff1017-30f4-4dd1-9a2e-46db9d70e57f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:53.771298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ff1017-30f4-4dd1-9a2e-46db9d70e57f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:53.791460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 56/113] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:44:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:53.793276 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:53.796803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ff1017-30f4-4dd1-9a2e-46db9d70e57f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:53.875708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:53.946889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ff1017-30f4-4dd1-9a2e-46db9d70e57f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:53.959356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbd168d-138a-4d56-b063-3b1185dfff08 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:53.995282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d697be3-1664-489c-81fe-b26c8477535c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:53.996486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d697be3-1664-489c-81fe-b26c8477535c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:53.996787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d697be3-1664-489c-81fe-b26c8477535c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:53.997077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d697be3-1664-489c-81fe-b26c8477535c None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:53.998296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d697be3-1664-489c-81fe-b26c8477535c None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:53.999771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d697be3-1664-489c-81fe-b26c8477535c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:54.001667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 58/114] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:53 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:54.002301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d697be3-1664-489c-81fe-b26c8477535c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:54.032156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191ac957-7a63-4f67-9b4f-3274c42c8217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:54.033094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191ac957-7a63-4f67-9b4f-3274c42c8217 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.033379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191ac957-7a63-4f67-9b4f-3274c42c8217 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.033658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191ac957-7a63-4f67-9b4f-3274c42c8217 None admin] PATH_INFO: `/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.038292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191ac957-7a63-4f67-9b4f-3274c42c8217 None admin] RBAC: Authorizing `identity:get_user(user_id=5c8b0f52ee8b4d4a82703c331b1da20e)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:54.039808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191ac957-7a63-4f67-9b4f-3274c42c8217 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:54.041619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 57/115] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:53 2022] GET /identity/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e => generated 275 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:54.042432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191ac957-7a63-4f67-9b4f-3274c42c8217 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:54.123647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:54.152941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d697be3-1664-489c-81fe-b26c8477535c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:54.186388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191ac957-7a63-4f67-9b4f-3274c42c8217 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:54.194482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c2465-0a75-4a8a-9bfd-33c8018379d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:54.195723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c2465-0a75-4a8a-9bfd-33c8018379d7 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.196009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c2465-0a75-4a8a-9bfd-33c8018379d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.196434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150c2465-0a75-4a8a-9bfd-33c8018379d7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.217722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150c2465-0a75-4a8a-9bfd-33c8018379d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-150c2465-0a75-4a8a-9bfd-33c8018379d7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:54.225582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:54.226507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 59/116] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:44:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:54.228603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c2465-0a75-4a8a-9bfd-33c8018379d7 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:54.235303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:54.244606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1680de46-ea63-46ea-83cc-e6c51b38791a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:54.245382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1680de46-ea63-46ea-83cc-e6c51b38791a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.248577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1680de46-ea63-46ea-83cc-e6c51b38791a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.248577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1680de46-ea63-46ea-83cc-e6c51b38791a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.248577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1680de46-ea63-46ea-83cc-e6c51b38791a None admin] RBAC: Adding query filter params (role.id=94339d681b5c4fd188991c590c897ad4, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569, user.id=aa2f0b25d6414893bb607fdcecb88286) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:54.248577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1680de46-ea63-46ea-83cc-e6c51b38791a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:54.249513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1680de46-ea63-46ea-83cc-e6c51b38791a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:54.255774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 58/117] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:44:54 2022] GET /identity/v3/role_assignments?user.id=aa2f0b25d6414893bb607fdcecb88286&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=94339d681b5c4fd188991c590c897ad4 => generated 259 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:54.257227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852318f1-c3a8-45ae-a384-35d411c2028a None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.257518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852318f1-c3a8-45ae-a384-35d411c2028a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.257781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852318f1-c3a8-45ae-a384-35d411c2028a None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.259174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 59/118] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:54.264627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852318f1-c3a8-45ae-a384-35d411c2028a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:54.366253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150c2465-0a75-4a8a-9bfd-33c8018379d7 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:54.384985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852318f1-c3a8-45ae-a384-35d411c2028a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:54.406758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dec61d5-a25f-48ca-8363-305a7cb76e44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:54.408122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec61d5-a25f-48ca-8363-305a7cb76e44 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.408616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec61d5-a25f-48ca-8363-305a7cb76e44 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.409171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dec61d5-a25f-48ca-8363-305a7cb76e44 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.410814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dec61d5-a25f-48ca-8363-305a7cb76e44 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:54.412705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dec61d5-a25f-48ca-8363-305a7cb76e44 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:54.414873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 60/119] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:54 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:54.416527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32c8cff-72f2-4f08-a962-d28920202e9c None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.417107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32c8cff-72f2-4f08-a962-d28920202e9c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.417587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32c8cff-72f2-4f08-a962-d28920202e9c None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.419219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 61/120] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:54.420711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d9024e-b5cf-4ce8-8184-d9c36556c5cf None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.421226 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d9024e-b5cf-4ce8-8184-d9c36556c5cf None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.421925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d9024e-b5cf-4ce8-8184-d9c36556c5cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.428323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b97cc2-f397-4dd9-ae76-535b14c14984 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:54.428323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b97cc2-f397-4dd9-ae76-535b14c14984 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.428323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b97cc2-f397-4dd9-ae76-535b14c14984 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.428323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b97cc2-f397-4dd9-ae76-535b14c14984 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.428685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b97cc2-f397-4dd9-ae76-535b14c14984 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:54.428685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b97cc2-f397-4dd9-ae76-535b14c14984 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:54.430527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b97cc2-f397-4dd9-ae76-535b14c14984 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:54.440533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 60/121] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:54.441841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-34d9024e-b5cf-4ce8-8184-d9c36556c5cf None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:54.443070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-34d9024e-b5cf-4ce8-8184-d9c36556c5cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:54.446263 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab5eff-a1a4-4777-9567-cf13757287a9 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.446822 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab5eff-a1a4-4777-9567-cf13757287a9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.447197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:54.447469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ab5eff-a1a4-4777-9567-cf13757287a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.465429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-61ab5eff-a1a4-4777-9567-cf13757287a9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:54.468360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-61ab5eff-a1a4-4777-9567-cf13757287a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:54.521709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 62/122] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:54.528452 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d9024e-b5cf-4ce8-8184-d9c36556c5cf None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:54.535291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:54.555987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 61/123] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:54.557090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ab5eff-a1a4-4777-9567-cf13757287a9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:54.677581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d9024e-b5cf-4ce8-8184-d9c36556c5cf None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:54.685837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ab5eff-a1a4-4777-9567-cf13757287a9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:54.713555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe42562-a39e-4f13-a178-71c7d35c3e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:54.714595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe42562-a39e-4f13-a178-71c7d35c3e5e None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.714895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe42562-a39e-4f13-a178-71c7d35c3e5e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.715173 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe42562-a39e-4f13-a178-71c7d35c3e5e None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/users/5c8b0f52ee8b4d4a82703c331b1da20e/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.717299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe42562-a39e-4f13-a178-71c7d35c3e5e None admin] RBAC: Authorizing `identity:create_grant(project_id=ceb60f2e4cad4cf29d7f7de724549569, user_id=5c8b0f52ee8b4d4a82703c331b1da20e, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:54.718933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe42562-a39e-4f13-a178-71c7d35c3e5e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:54.723203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b4d7dd-64e6-45b9-98a8-3cc5b365e5a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:54.724613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b4d7dd-64e6-45b9-98a8-3cc5b365e5a5 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.724979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b4d7dd-64e6-45b9-98a8-3cc5b365e5a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.725361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b4d7dd-64e6-45b9-98a8-3cc5b365e5a5 None admin] PATH_INFO: `/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.726758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b4d7dd-64e6-45b9-98a8-3cc5b365e5a5 None admin] RBAC: Authorizing `identity:get_user(user_id=05ecde7b5ffd440ea8bdbb9c0c6418c1)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:54.728332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b4d7dd-64e6-45b9-98a8-3cc5b365e5a5 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:54.730876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 62/124] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:54 2022] GET /identity/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1 => generated 247 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:54.731580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b4d7dd-64e6-45b9-98a8-3cc5b365e5a5 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:54.789188 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 63/125] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:44:54 2022] PUT /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/users/5c8b0f52ee8b4d4a82703c331b1da20e/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:44:54.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe42562-a39e-4f13-a178-71c7d35c3e5e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:54.791364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:54.809202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:54.892519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b4d7dd-64e6-45b9-98a8-3cc5b365e5a5 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:54.940265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe42562-a39e-4f13-a178-71c7d35c3e5e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:54.964619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f01484-1f70-405c-8e49-266b91b1b01a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:54.965482 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f01484-1f70-405c-8e49-266b91b1b01a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.965775 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f01484-1f70-405c-8e49-266b91b1b01a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.966060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09f01484-1f70-405c-8e49-266b91b1b01a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.987766 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1514e5-fe38-4cbd-b49a-1a11543c4a78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:54.989161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1514e5-fe38-4cbd-b49a-1a11543c4a78 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:54.989672 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1514e5-fe38-4cbd-b49a-1a11543c4a78 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:54.990264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1514e5-fe38-4cbd-b49a-1a11543c4a78 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:54.994172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09f01484-1f70-405c-8e49-266b91b1b01a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-09f01484-1f70-405c-8e49-266b91b1b01a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:55.003953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:55.011040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:44:55.011972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:44:55.012547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1514e5-fe38-4cbd-b49a-1a11543c4a78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:55.012624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 63/126] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:44:54 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 275 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:55.012624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f01484-1f70-405c-8e49-266b91b1b01a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-2e1514e5-fe38-4cbd-b49a-1a11543c4a78 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:55.015525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:55.016468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 64/127] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:54 2022] GET /identity/v3/roles/member => generated 84 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:55.017279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1514e5-fe38-4cbd-b49a-1a11543c4a78 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:55.095983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:55.144288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1514e5-fe38-4cbd-b49a-1a11543c4a78 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:55.183852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4e2414-8c5d-44fd-a020-ce459b65eef2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:55.185655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4e2414-8c5d-44fd-a020-ce459b65eef2 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.186472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4e2414-8c5d-44fd-a020-ce459b65eef2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.187274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a4e2414-8c5d-44fd-a020-ce459b65eef2 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.188894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4e2414-8c5d-44fd-a020-ce459b65eef2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:55.189434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4e2414-8c5d-44fd-a020-ce459b65eef2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:55.191068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a4e2414-8c5d-44fd-a020-ce459b65eef2 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:55.200307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 65/128] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:44:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:55.205736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4e2414-8c5d-44fd-a020-ce459b65eef2 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:55.236211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09f01484-1f70-405c-8e49-266b91b1b01a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:55.249815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:55.271641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a3c2f6-10e1-43b9-8a8c-3addf8e30d88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:55.272538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a3c2f6-10e1-43b9-8a8c-3addf8e30d88 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.272890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a3c2f6-10e1-43b9-8a8c-3addf8e30d88 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.273126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a3c2f6-10e1-43b9-8a8c-3addf8e30d88 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.274334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a3c2f6-10e1-43b9-8a8c-3addf8e30d88 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:55.276023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a3c2f6-10e1-43b9-8a8c-3addf8e30d88 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:55.283241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 64/129] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:55 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:55.284100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a3c2f6-10e1-43b9-8a8c-3addf8e30d88 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:55.351035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a4e2414-8c5d-44fd-a020-ce459b65eef2 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:55.379154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:55.390785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a3c2f6-10e1-43b9-8a8c-3addf8e30d88 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:55.395193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1350a9e-fab0-461b-b8ed-85e53890b994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:55.395951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1350a9e-fab0-461b-b8ed-85e53890b994 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.396321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1350a9e-fab0-461b-b8ed-85e53890b994 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.396683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1350a9e-fab0-461b-b8ed-85e53890b994 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.397548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1350a9e-fab0-461b-b8ed-85e53890b994 None admin] RBAC: Adding query filter params (name=member) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:55.397932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1350a9e-fab0-461b-b8ed-85e53890b994 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:55.399642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1350a9e-fab0-461b-b8ed-85e53890b994 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:55.410175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 66/130] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:44:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:55.411159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1350a9e-fab0-461b-b8ed-85e53890b994 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:55.429790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19323bc-ff97-4070-81ab-e2b2cc33277c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:55.431046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19323bc-ff97-4070-81ab-e2b2cc33277c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.431610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19323bc-ff97-4070-81ab-e2b2cc33277c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.432063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19323bc-ff97-4070-81ab-e2b2cc33277c None admin] PATH_INFO: `/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.433557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19323bc-ff97-4070-81ab-e2b2cc33277c None admin] RBAC: Authorizing `identity:get_user(user_id=5c8b0f52ee8b4d4a82703c331b1da20e)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:55.435410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19323bc-ff97-4070-81ab-e2b2cc33277c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:55.437271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 65/131] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:55 2022] GET /identity/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e => generated 275 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:55.438376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19323bc-ff97-4070-81ab-e2b2cc33277c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:55.491090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:55.530485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1350a9e-fab0-461b-b8ed-85e53890b994 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:55.546160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19323bc-ff97-4070-81ab-e2b2cc33277c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:55.565860 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b66ab5a-1bcf-42a7-9d63-aad5b88bf628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:55.566761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b66ab5a-1bcf-42a7-9d63-aad5b88bf628 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.567052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b66ab5a-1bcf-42a7-9d63-aad5b88bf628 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.567396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b66ab5a-1bcf-42a7-9d63-aad5b88bf628 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/users/05ecde7b5ffd440ea8bdbb9c0c6418c1/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.574558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b66ab5a-1bcf-42a7-9d63-aad5b88bf628 None admin] RBAC: Authorizing `identity:create_grant(project_id=ceb60f2e4cad4cf29d7f7de724549569, user_id=05ecde7b5ffd440ea8bdbb9c0c6418c1, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:55.574558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b66ab5a-1bcf-42a7-9d63-aad5b88bf628 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:55.593175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1d5f1f-38bf-45c7-9b91-803a78a8382b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:55.594158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1d5f1f-38bf-45c7-9b91-803a78a8382b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.594447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1d5f1f-38bf-45c7-9b91-803a78a8382b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.594731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1d5f1f-38bf-45c7-9b91-803a78a8382b None admin] PATH_INFO: `/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.596132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1d5f1f-38bf-45c7-9b91-803a78a8382b None admin] RBAC: Authorizing `identity:get_user(user_id=5c8b0f52ee8b4d4a82703c331b1da20e)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:55.597648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1d5f1f-38bf-45c7-9b91-803a78a8382b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:55.599729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 66/132] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:55 2022] GET /identity/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e => generated 275 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:55.600602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1d5f1f-38bf-45c7-9b91-803a78a8382b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:55.647398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 67/133] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:44:55 2022] PUT /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/users/05ecde7b5ffd440ea8bdbb9c0c6418c1/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:44:55.648051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b66ab5a-1bcf-42a7-9d63-aad5b88bf628 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:55.737140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:55.760539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1d5f1f-38bf-45c7-9b91-803a78a8382b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:55.794794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:55.823354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c013b6d-98f8-4829-9552-10b1b46b1c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:55.824381 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c013b6d-98f8-4829-9552-10b1b46b1c46 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.824676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c013b6d-98f8-4829-9552-10b1b46b1c46 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.825036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c013b6d-98f8-4829-9552-10b1b46b1c46 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.828639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c013b6d-98f8-4829-9552-10b1b46b1c46 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:55.830330 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c013b6d-98f8-4829-9552-10b1b46b1c46 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:55.832423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 67/134] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:55 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:55.833649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae1696-5a46-496e-84a9-017ed5eba648 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.833945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae1696-5a46-496e-84a9-017ed5eba648 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.834232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaae1696-5a46-496e-84a9-017ed5eba648 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.835897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 68/135] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:55.842187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a376f4b-a7b3-4621-a55e-4173af16401d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.842484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a376f4b-a7b3-4621-a55e-4173af16401d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.842772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a376f4b-a7b3-4621-a55e-4173af16401d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.865159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-1a376f4b-a7b3-4621-a55e-4173af16401d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:55.866996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1a376f4b-a7b3-4621-a55e-4173af16401d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:55.906738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b66ab5a-1bcf-42a7-9d63-aad5b88bf628 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:55.949465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 69/136] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:44:55.950378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a376f4b-a7b3-4621-a55e-4173af16401d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:55.965435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5409171a-2b93-4621-bb83-436a1da9abf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:55.966405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5409171a-2b93-4621-bb83-436a1da9abf5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:55.966771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5409171a-2b93-4621-bb83-436a1da9abf5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:55.967147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5409171a-2b93-4621-bb83-436a1da9abf5 None admin] PATH_INFO: `/v3/projects/edf1e3eb7289421a9782aa94730b2021` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:55.969328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5409171a-2b93-4621-bb83-436a1da9abf5 None admin] RBAC: Authorizing `identity:get_project(project_id=edf1e3eb7289421a9782aa94730b2021)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:55.971013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5409171a-2b93-4621-bb83-436a1da9abf5 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:55.973425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 68/137] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:55 2022] GET /identity/v3/projects/edf1e3eb7289421a9782aa94730b2021 => generated 312 bytes in 326 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:55.981739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5409171a-2b93-4621-bb83-436a1da9abf5 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:56.078951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a376f4b-a7b3-4621-a55e-4173af16401d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:56.084569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5409171a-2b93-4621-bb83-436a1da9abf5 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:56.096804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:56.097759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:56.120638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3142212-38c0-47de-bbab-552fae5b9f6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:56.122040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3142212-38c0-47de-bbab-552fae5b9f6a None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:56.122575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3142212-38c0-47de-bbab-552fae5b9f6a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:56.123049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3142212-38c0-47de-bbab-552fae5b9f6a None admin] PATH_INFO: `/v3/projects/edf1e3eb7289421a9782aa94730b2021/users/5c8b0f52ee8b4d4a82703c331b1da20e/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:56.125113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a973cd7-ea58-4855-86c7-90c928c7686e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:56.125421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a973cd7-ea58-4855-86c7-90c928c7686e None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:56.125714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a973cd7-ea58-4855-86c7-90c928c7686e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:56.126002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a973cd7-ea58-4855-86c7-90c928c7686e None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/users/5c8b0f52ee8b4d4a82703c331b1da20e/roles/df56ac8b9c6f46958342cc16f10cea63` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:56.127197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3142212-38c0-47de-bbab-552fae5b9f6a None admin] RBAC: Authorizing `identity:create_grant(project_id=edf1e3eb7289421a9782aa94730b2021, user_id=5c8b0f52ee8b4d4a82703c331b1da20e, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:56.128946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3142212-38c0-47de-bbab-552fae5b9f6a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:56.138599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a973cd7-ea58-4855-86c7-90c928c7686e None admin] RBAC: Authorizing `identity:create_grant(project_id=ceb60f2e4cad4cf29d7f7de724549569, user_id=5c8b0f52ee8b4d4a82703c331b1da20e, role_id=df56ac8b9c6f46958342cc16f10cea63)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:56.140445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a973cd7-ea58-4855-86c7-90c928c7686e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:56.220381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 69/138] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:44:55 2022] PUT /identity/v3/projects/edf1e3eb7289421a9782aa94730b2021/users/5c8b0f52ee8b4d4a82703c331b1da20e/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:44:56.222088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3142212-38c0-47de-bbab-552fae5b9f6a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:56.250805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 70/139] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:44:55 2022] PUT /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/users/5c8b0f52ee8b4d4a82703c331b1da20e/roles/df56ac8b9c6f46958342cc16f10cea63 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:44:56.352896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3142212-38c0-47de-bbab-552fae5b9f6a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:56.400260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837f9887-7c39-46d2-a2aa-744d73e9124b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:56.401135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837f9887-7c39-46d2-a2aa-744d73e9124b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:56.401473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837f9887-7c39-46d2-a2aa-744d73e9124b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:56.401764 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837f9887-7c39-46d2-a2aa-744d73e9124b None admin] PATH_INFO: `/v3/domains/default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:56.403192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837f9887-7c39-46d2-a2aa-744d73e9124b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:56.405064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837f9887-7c39-46d2-a2aa-744d73e9124b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:56.406934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 70/140] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:44:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:56.412604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a973cd7-ea58-4855-86c7-90c928c7686e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:56.541129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a973cd7-ea58-4855-86c7-90c928c7686e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:56.577375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e6d6dd1-3e17-4d75-bef4-86a8ab7fe7ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:56.578055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6d6dd1-3e17-4d75-bef4-86a8ab7fe7ca None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:56.578344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6d6dd1-3e17-4d75-bef4-86a8ab7fe7ca None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:56.578627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e6d6dd1-3e17-4d75-bef4-86a8ab7fe7ca None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:56.579370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6d6dd1-3e17-4d75-bef4-86a8ab7fe7ca None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:56.580860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e6d6dd1-3e17-4d75-bef4-86a8ab7fe7ca None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:56.675461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 71/141] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:44:56 2022] POST /identity/v3/users => generated 283 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:44:56.937175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1bec24-d89c-4816-9a32-3073801518a8 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:56.937522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1bec24-d89c-4816-9a32-3073801518a8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:56.937711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1bec24-d89c-4816-9a32-3073801518a8 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:56.939036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 71/142] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:44:56.944678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec41e0f-5831-47fc-9884-6c7106dc39f3 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:56.945003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec41e0f-5831-47fc-9884-6c7106dc39f3 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:56.945358 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec41e0f-5831-47fc-9884-6c7106dc39f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:56.968967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5ec41e0f-5831-47fc-9884-6c7106dc39f3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:56.970226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5ec41e0f-5831-47fc-9884-6c7106dc39f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:57.002609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:57.063257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 72/143] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:57.110858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1bec24-d89c-4816-9a32-3073801518a8 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:57.166429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:57.247436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1bec24-d89c-4816-9a32-3073801518a8 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:57.288271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd546ba-c0e6-488f-815c-c0df29b6a0ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:57.289138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd546ba-c0e6-488f-815c-c0df29b6a0ef None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:57.289425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd546ba-c0e6-488f-815c-c0df29b6a0ef None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:57.289705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd546ba-c0e6-488f-815c-c0df29b6a0ef None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:57.303560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd546ba-c0e6-488f-815c-c0df29b6a0ef None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-cdd546ba-c0e6-488f-815c-c0df29b6a0ef None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:57.311833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:57.312680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:57.313455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 72/144] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:57 2022] GET /identity/v3/roles/reader => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:57.325934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec41e0f-5831-47fc-9884-6c7106dc39f3 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:57.453020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec41e0f-5831-47fc-9884-6c7106dc39f3 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:57.500351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9a8330-4779-45e1-b541-f47dd095de05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:57.500989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9a8330-4779-45e1-b541-f47dd095de05 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:57.501279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9a8330-4779-45e1-b541-f47dd095de05 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:57.501557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa9a8330-4779-45e1-b541-f47dd095de05 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:57.502330 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9a8330-4779-45e1-b541-f47dd095de05 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:57.502634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9a8330-4779-45e1-b541-f47dd095de05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:57.504043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa9a8330-4779-45e1-b541-f47dd095de05 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:57.519262 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 73/145] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:44:57 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:57.524936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd546ba-c0e6-488f-815c-c0df29b6a0ef None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:57.709730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:57.713347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd546ba-c0e6-488f-815c-c0df29b6a0ef None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:57.754113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a83403d-7dbf-42ee-804d-33300b09585c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:57.754923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a83403d-7dbf-42ee-804d-33300b09585c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:57.755210 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a83403d-7dbf-42ee-804d-33300b09585c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:57.755491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a83403d-7dbf-42ee-804d-33300b09585c None admin] PATH_INFO: `/v3/users/aa2f0b25d6414893bb607fdcecb88286` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:57.756722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a83403d-7dbf-42ee-804d-33300b09585c None admin] RBAC: Authorizing `identity:get_user(user_id=aa2f0b25d6414893bb607fdcecb88286)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:57.758249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a83403d-7dbf-42ee-804d-33300b09585c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:57.760331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 73/146] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:57 2022] GET /identity/v3/users/aa2f0b25d6414893bb607fdcecb88286 => generated 289 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:57.767781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9a8330-4779-45e1-b541-f47dd095de05 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:57.811430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:57.903406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a294f5-aec5-4958-a45a-2a674b238948 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:57.903721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a294f5-aec5-4958-a45a-2a674b238948 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:57.903977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a294f5-aec5-4958-a45a-2a674b238948 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:57.905825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 74/147] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:44:57.911160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ad6a8-d9a4-4878-af24-68d7126e8fce None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:57.911449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ad6a8-d9a4-4878-af24-68d7126e8fce None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:57.911734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4ad6a8-d9a4-4878-af24-68d7126e8fce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:57.933807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1d4ad6a8-d9a4-4878-af24-68d7126e8fce None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:57.934945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa9a8330-4779-45e1-b541-f47dd095de05 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:57.935458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1d4ad6a8-d9a4-4878-af24-68d7126e8fce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:57.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f02fe3-7ba6-4cb4-a26c-0aac5423a233 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:57.989864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f02fe3-7ba6-4cb4-a26c-0aac5423a233 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:57.990452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f02fe3-7ba6-4cb4-a26c-0aac5423a233 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:57.990931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f02fe3-7ba6-4cb4-a26c-0aac5423a233 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:57.995280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f02fe3-7ba6-4cb4-a26c-0aac5423a233 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:57.997602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f02fe3-7ba6-4cb4-a26c-0aac5423a233 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:57.999797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 74/148] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:57 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:58.005865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f02fe3-7ba6-4cb4-a26c-0aac5423a233 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:58.046289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 75/149] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:44:58.088825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4ad6a8-d9a4-4878-af24-68d7126e8fce None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:58.155664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:58.173752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f02fe3-7ba6-4cb4-a26c-0aac5423a233 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:58.233505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d260cbc8-c6ce-494b-8275-b33a494cfcb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:58.234826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d260cbc8-c6ce-494b-8275-b33a494cfcb4 None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:58.235412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d260cbc8-c6ce-494b-8275-b33a494cfcb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:58.235891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d260cbc8-c6ce-494b-8275-b33a494cfcb4 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/users/aa2f0b25d6414893bb607fdcecb88286/roles/94339d681b5c4fd188991c590c897ad4` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:58.239130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d260cbc8-c6ce-494b-8275-b33a494cfcb4 None admin] RBAC: Authorizing `identity:create_grant(project_id=ceb60f2e4cad4cf29d7f7de724549569, user_id=aa2f0b25d6414893bb607fdcecb88286, role_id=94339d681b5c4fd188991c590c897ad4)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:58.240924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d260cbc8-c6ce-494b-8275-b33a494cfcb4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:58.295113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:58.300910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4ad6a8-d9a4-4878-af24-68d7126e8fce None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:58.336468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 75/150] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:44:58 2022] PUT /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/users/aa2f0b25d6414893bb607fdcecb88286/roles/94339d681b5c4fd188991c590c897ad4 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 12:44:58.345023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27231569-177b-4083-8285-3d4dbe095857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:58.345956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27231569-177b-4083-8285-3d4dbe095857 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:58.346313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27231569-177b-4083-8285-3d4dbe095857 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:58.346693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27231569-177b-4083-8285-3d4dbe095857 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:58.361074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27231569-177b-4083-8285-3d4dbe095857 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-27231569-177b-4083-8285-3d4dbe095857 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:58.369185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:58.370124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:58.370979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 76/151] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:44:58.376349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d260cbc8-c6ce-494b-8275-b33a494cfcb4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:58.488558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d260cbc8-c6ce-494b-8275-b33a494cfcb4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:58.525552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2d66cc-3c9e-45dc-a975-cb132f3874e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:58.526230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2d66cc-3c9e-45dc-a975-cb132f3874e3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:58.526513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2d66cc-3c9e-45dc-a975-cb132f3874e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:58.526792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2d66cc-3c9e-45dc-a975-cb132f3874e3 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:58.527624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2d66cc-3c9e-45dc-a975-cb132f3874e3 None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:58.527984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2d66cc-3c9e-45dc-a975-cb132f3874e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:58.529665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2d66cc-3c9e-45dc-a975-cb132f3874e3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:58.538842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 76/152] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:44:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:58.545596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27231569-177b-4083-8285-3d4dbe095857 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:58.660023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27231569-177b-4083-8285-3d4dbe095857 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:58.704300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983d647e-5a4b-4d5b-a066-cd44b357654e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:58.705372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d647e-5a4b-4d5b-a066-cd44b357654e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:58.705654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d647e-5a4b-4d5b-a066-cd44b357654e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:58.705978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d647e-5a4b-4d5b-a066-cd44b357654e None admin] PATH_INFO: `/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:58.713851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983d647e-5a4b-4d5b-a066-cd44b357654e None admin] RBAC: Authorizing `identity:get_user(user_id=5c8b0f52ee8b4d4a82703c331b1da20e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:58.715463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983d647e-5a4b-4d5b-a066-cd44b357654e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:58.717426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 77/153] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:44:58 2022] GET /identity/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e => generated 275 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:58.722788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2d66cc-3c9e-45dc-a975-cb132f3874e3 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:58.739537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:58.835730 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2d66cc-3c9e-45dc-a975-cb132f3874e3 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:58.871489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db21233b-872a-42e6-ab47-68ad11c261a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:58.872142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db21233b-872a-42e6-ab47-68ad11c261a4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:58.872420 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db21233b-872a-42e6-ab47-68ad11c261a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:58.872693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db21233b-872a-42e6-ab47-68ad11c261a4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:58.883060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e9f60c-5811-428f-bd7e-9a1ab772da51 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:58.883337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e9f60c-5811-428f-bd7e-9a1ab772da51 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:58.883620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e9f60c-5811-428f-bd7e-9a1ab772da51 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:58.885215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 78/154] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:44:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:44:58.890041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ad7bc3-7683-4c10-95fa-1a838c7667c8 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:58.890324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ad7bc3-7683-4c10-95fa-1a838c7667c8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:58.890605 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ad7bc3-7683-4c10-95fa-1a838c7667c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:58.917862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-23ad7bc3-7683-4c10-95fa-1a838c7667c8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:44:58.919063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-23ad7bc3-7683-4c10-95fa-1a838c7667c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:44:58.972450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db21233b-872a-42e6-ab47-68ad11c261a4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:58.973954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:58.974140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db21233b-872a-42e6-ab47-68ad11c261a4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:59.001168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:59.131268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 77/155] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:44:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:44:59.133716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 79/156] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:44:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:44:59.137122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db21233b-872a-42e6-ab47-68ad11c261a4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:59.185351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ad7bc3-7683-4c10-95fa-1a838c7667c8 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:59.285751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db21233b-872a-42e6-ab47-68ad11c261a4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:59.317609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:59.366509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1ed38a-5376-4dd6-840f-58dc1a13a7e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:59.368817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1ed38a-5376-4dd6-840f-58dc1a13a7e4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:59.369340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1ed38a-5376-4dd6-840f-58dc1a13a7e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:59.369766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1ed38a-5376-4dd6-840f-58dc1a13a7e4 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:59.371774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1ed38a-5376-4dd6-840f-58dc1a13a7e4 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:59.373420 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b1ed38a-5376-4dd6-840f-58dc1a13a7e4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:59.376241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 78/157] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:44:59 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:44:59.382237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1ed38a-5376-4dd6-840f-58dc1a13a7e4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:59.544794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1ed38a-5376-4dd6-840f-58dc1a13a7e4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:59.604368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6690b689-3e33-41b5-bd2c-b9e1ce6badf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:59.605457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6690b689-3e33-41b5-bd2c-b9e1ce6badf0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:59.605929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6690b689-3e33-41b5-bd2c-b9e1ce6badf0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:59.606377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6690b689-3e33-41b5-bd2c-b9e1ce6badf0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:59.607397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6690b689-3e33-41b5-bd2c-b9e1ce6badf0 None admin] RBAC: Adding query filter params (role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569, user.id=5c8b0f52ee8b4d4a82703c331b1da20e) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:59.607878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6690b689-3e33-41b5-bd2c-b9e1ce6badf0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:59.609536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6690b689-3e33-41b5-bd2c-b9e1ce6badf0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:59.620203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 79/158] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:44:59 2022] GET /identity/v3/role_assignments?user.id=5c8b0f52ee8b4d4a82703c331b1da20e&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 608 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:44:59.623666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ad7bc3-7683-4c10-95fa-1a838c7667c8 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:59.707293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c5600d-1796-48b8-bcbc-4d8c231c01a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:59.708114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5600d-1796-48b8-bcbc-4d8c231c01a3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:59.708397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5600d-1796-48b8-bcbc-4d8c231c01a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:59.708676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74c5600d-1796-48b8-bcbc-4d8c231c01a3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:59.730906 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74c5600d-1796-48b8-bcbc-4d8c231c01a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-74c5600d-1796-48b8-bcbc-4d8c231c01a3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:44:59.742956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:44:59.743809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:44:59.746279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 80/159] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:44:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 559 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:44:59.752082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6690b689-3e33-41b5-bd2c-b9e1ce6badf0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:44:59.849056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:44:59.930519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6690b689-3e33-41b5-bd2c-b9e1ce6badf0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:44:59.980562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cd3b4b-7a3d-4cd5-911a-cd084200373c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:44:59.981195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cd3b4b-7a3d-4cd5-911a-cd084200373c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:44:59.981480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cd3b4b-7a3d-4cd5-911a-cd084200373c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:44:59.981753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01cd3b4b-7a3d-4cd5-911a-cd084200373c None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:44:59.982501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cd3b4b-7a3d-4cd5-911a-cd084200373c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:44:59.982804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cd3b4b-7a3d-4cd5-911a-cd084200373c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:44:59.984196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01cd3b4b-7a3d-4cd5-911a-cd084200373c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:44:59.997117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 80/160] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:44:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:00.005853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c5600d-1796-48b8-bcbc-4d8c231c01a3 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:00.029720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:00.135417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74c5600d-1796-48b8-bcbc-4d8c231c01a3 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:00.175861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b4b231-b109-4a92-88d9-b64b9743ef58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:00.176758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b4b231-b109-4a92-88d9-b64b9743ef58 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:00.177064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b4b231-b109-4a92-88d9-b64b9743ef58 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:00.177343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b4b231-b109-4a92-88d9-b64b9743ef58 None admin] PATH_INFO: `/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:00.178570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b4b231-b109-4a92-88d9-b64b9743ef58 None admin] RBAC: Authorizing `identity:get_user(user_id=05ecde7b5ffd440ea8bdbb9c0c6418c1)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:00.180048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b4b231-b109-4a92-88d9-b64b9743ef58 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:00.181888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 81/161] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:00 2022] GET /identity/v3/users/05ecde7b5ffd440ea8bdbb9c0c6418c1 => generated 247 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:00.187196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cd3b4b-7a3d-4cd5-911a-cd084200373c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:00.384688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01cd3b4b-7a3d-4cd5-911a-cd084200373c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:00.391183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5953a22b-6f43-4f7f-bb9e-8c4d538328a5 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:00.391478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5953a22b-6f43-4f7f-bb9e-8c4d538328a5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:00.391768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5953a22b-6f43-4f7f-bb9e-8c4d538328a5 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:00.391929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:00.397520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 82/162] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:00 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:00.403166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d9218-e74b-4da3-ab1a-eb2d27d81549 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:00.407971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d9218-e74b-4da3-ab1a-eb2d27d81549 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:00.407971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907d9218-e74b-4da3-ab1a-eb2d27d81549 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:00.426703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b525cfb3-ded2-4f50-8656-a9b34cfb18b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:00.427360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525cfb3-ded2-4f50-8656-a9b34cfb18b4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:00.427638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525cfb3-ded2-4f50-8656-a9b34cfb18b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:00.427910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b525cfb3-ded2-4f50-8656-a9b34cfb18b4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:00.435023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-907d9218-e74b-4da3-ab1a-eb2d27d81549 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:00.436192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-907d9218-e74b-4da3-ab1a-eb2d27d81549 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:00.546116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b525cfb3-ded2-4f50-8656-a9b34cfb18b4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:00.547642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b525cfb3-ded2-4f50-8656-a9b34cfb18b4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:00.581045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 83/163] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:00.639647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d9218-e74b-4da3-ab1a-eb2d27d81549 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:00.669192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:00.716429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 81/164] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 530 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:00.717638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8103991-00f6-4aa9-9050-e50c4449c85d None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:00.717926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8103991-00f6-4aa9-9050-e50c4449c85d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:00.718195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8103991-00f6-4aa9-9050-e50c4449c85d None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:00.719640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 82/165] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:00.725272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8103991-00f6-4aa9-9050-e50c4449c85d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:00.741446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:00.850413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907d9218-e74b-4da3-ab1a-eb2d27d81549 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:00.915542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8103991-00f6-4aa9-9050-e50c4449c85d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:00.915729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd5437f-deda-44fa-b2eb-4a78f06aa973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:00.916314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd5437f-deda-44fa-b2eb-4a78f06aa973 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:00.916601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd5437f-deda-44fa-b2eb-4a78f06aa973 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:00.916899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd5437f-deda-44fa-b2eb-4a78f06aa973 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:00.941712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd5437f-deda-44fa-b2eb-4a78f06aa973 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-bcd5437f-deda-44fa-b2eb-4a78f06aa973 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:00.956359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:00.960266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 84/166] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:00 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 319 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:00.961268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e82f67-a071-4f6d-9779-4f3725dd0d05 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:00.961968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e82f67-a071-4f6d-9779-4f3725dd0d05 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:00.961968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e82f67-a071-4f6d-9779-4f3725dd0d05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:00.963099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77aee46-7ee4-42f6-a157-fce81e898aea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:00.964239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77aee46-7ee4-42f6-a157-fce81e898aea None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:00.964706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77aee46-7ee4-42f6-a157-fce81e898aea None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:00.965166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77aee46-7ee4-42f6-a157-fce81e898aea None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:00.966590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77aee46-7ee4-42f6-a157-fce81e898aea None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:00.968234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77aee46-7ee4-42f6-a157-fce81e898aea None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:00.974920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 83/167] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:00 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:00.976042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77aee46-7ee4-42f6-a157-fce81e898aea None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:01.000984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e7e82f67-a071-4f6d-9779-4f3725dd0d05 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:01.002176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e7e82f67-a071-4f6d-9779-4f3725dd0d05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:01.063637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:01.089098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77aee46-7ee4-42f6-a157-fce81e898aea None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:01.124385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd830822-462b-47fe-870e-557a292c5e97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:01.125516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd830822-462b-47fe-870e-557a292c5e97 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:01.126037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd830822-462b-47fe-870e-557a292c5e97 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:01.126553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd830822-462b-47fe-870e-557a292c5e97 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:01.127504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd830822-462b-47fe-870e-557a292c5e97 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:01.128000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd830822-462b-47fe-870e-557a292c5e97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:01.129625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd830822-462b-47fe-870e-557a292c5e97 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:01.138229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 84/168] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:00 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:01.139243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd830822-462b-47fe-870e-557a292c5e97 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:01.150684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 85/169] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:01.151542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e82f67-a071-4f6d-9779-4f3725dd0d05 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:01.154701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:01.241900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd830822-462b-47fe-870e-557a292c5e97 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:01.301703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892238bb-d127-48ef-94db-cf9d4d3743d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:01.303946 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892238bb-d127-48ef-94db-cf9d4d3743d3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:01.304454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892238bb-d127-48ef-94db-cf9d4d3743d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:01.305094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-892238bb-d127-48ef-94db-cf9d4d3743d3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:01.307364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892238bb-d127-48ef-94db-cf9d4d3743d3 None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569, user.id=05ecde7b5ffd440ea8bdbb9c0c6418c1) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:01.308642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892238bb-d127-48ef-94db-cf9d4d3743d3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:01.310361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-892238bb-d127-48ef-94db-cf9d4d3743d3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:01.322418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 85/170] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:01 2022] GET /identity/v3/role_assignments?user.id=05ecde7b5ffd440ea8bdbb9c0c6418c1&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=766799d1b6c049168590571f1a959b3b => generated 608 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:01.324294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892238bb-d127-48ef-94db-cf9d4d3743d3 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:01.327160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:01.335670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e82f67-a071-4f6d-9779-4f3725dd0d05 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:01.385433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c0d320-d8d1-42bf-83b0-16dfe1ed6bee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:01.387472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c0d320-d8d1-42bf-83b0-16dfe1ed6bee None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:01.387985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c0d320-d8d1-42bf-83b0-16dfe1ed6bee None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:01.388544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c0d320-d8d1-42bf-83b0-16dfe1ed6bee None admin] PATH_INFO: `/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:01.390161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c0d320-d8d1-42bf-83b0-16dfe1ed6bee None admin] RBAC: Authorizing `identity:get_user(user_id=5c8b0f52ee8b4d4a82703c331b1da20e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:01.391978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c0d320-d8d1-42bf-83b0-16dfe1ed6bee None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:01.394548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 86/171] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:01 2022] GET /identity/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e => generated 275 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:01.402589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c0d320-d8d1-42bf-83b0-16dfe1ed6bee None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:01.526571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:01.558413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c0d320-d8d1-42bf-83b0-16dfe1ed6bee None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:01.601427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-892238bb-d127-48ef-94db-cf9d4d3743d3 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:01.612038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c96a6d-a2fc-4ee0-94e9-198f6d190812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:01.617268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c96a6d-a2fc-4ee0-94e9-198f6d190812 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:01.617776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c96a6d-a2fc-4ee0-94e9-198f6d190812 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:01.618351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c96a6d-a2fc-4ee0-94e9-198f6d190812 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:01.670065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880f2d82-14e3-4ffb-99b7-c81fe65af81b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:01.670921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880f2d82-14e3-4ffb-99b7-c81fe65af81b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:01.671205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880f2d82-14e3-4ffb-99b7-c81fe65af81b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:01.671480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880f2d82-14e3-4ffb-99b7-c81fe65af81b None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:01.701737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880f2d82-14e3-4ffb-99b7-c81fe65af81b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-880f2d82-14e3-4ffb-99b7-c81fe65af81b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:01.717621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:01.718645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:01.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 86/172] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 395 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:01.720251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956b18a1-eb13-4d0d-973a-6ba04bfb5cb7 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:01.721664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956b18a1-eb13-4d0d-973a-6ba04bfb5cb7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:01.721664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956b18a1-eb13-4d0d-973a-6ba04bfb5cb7 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:01.722799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 87/173] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:01.727698 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956b18a1-eb13-4d0d-973a-6ba04bfb5cb7 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:01.731823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c96a6d-a2fc-4ee0-94e9-198f6d190812 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:01.733384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c96a6d-a2fc-4ee0-94e9-198f6d190812 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:01.835514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:01.861012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:01.892076 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 87/174] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:01.898697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813286d4-c6cf-40bc-8660-4f0f78f539e3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:01.898996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813286d4-c6cf-40bc-8660-4f0f78f539e3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:01.899278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813286d4-c6cf-40bc-8660-4f0f78f539e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:01.928405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-813286d4-c6cf-40bc-8660-4f0f78f539e3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:01.930308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-813286d4-c6cf-40bc-8660-4f0f78f539e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:01.942593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956b18a1-eb13-4d0d-973a-6ba04bfb5cb7 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:02.006117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897d18f6-1812-4da5-8627-ef6c6a62479a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:02.006801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d18f6-1812-4da5-8627-ef6c6a62479a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.007083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d18f6-1812-4da5-8627-ef6c6a62479a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.007363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897d18f6-1812-4da5-8627-ef6c6a62479a None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.008113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897d18f6-1812-4da5-8627-ef6c6a62479a None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:02.008417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897d18f6-1812-4da5-8627-ef6c6a62479a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:02.009834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897d18f6-1812-4da5-8627-ef6c6a62479a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:02.026366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 88/175] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:02.030078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897d18f6-1812-4da5-8627-ef6c6a62479a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:02.060903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 88/176] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:02.062513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813286d4-c6cf-40bc-8660-4f0f78f539e3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:02.204624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897d18f6-1812-4da5-8627-ef6c6a62479a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:02.222900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:02.225857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813286d4-c6cf-40bc-8660-4f0f78f539e3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:02.236226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:02.253212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb4d83-c686-4493-8188-8b08cd6cd2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:02.254429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb4d83-c686-4493-8188-8b08cd6cd2ff None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.254777 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb4d83-c686-4493-8188-8b08cd6cd2ff None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.255301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb4d83-c686-4493-8188-8b08cd6cd2ff None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.257200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb4d83-c686-4493-8188-8b08cd6cd2ff None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:02.259235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb4d83-c686-4493-8188-8b08cd6cd2ff None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:02.262144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 89/177] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:02 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:02.263377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb4d83-c686-4493-8188-8b08cd6cd2ff None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:02.269668 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7253ee9d-5f4c-4959-b046-b1544d1839a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:02.270559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7253ee9d-5f4c-4959-b046-b1544d1839a1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.270845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7253ee9d-5f4c-4959-b046-b1544d1839a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.271122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7253ee9d-5f4c-4959-b046-b1544d1839a1 None admin] PATH_INFO: `/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.272342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7253ee9d-5f4c-4959-b046-b1544d1839a1 None admin] RBAC: Authorizing `identity:get_user(user_id=5c8b0f52ee8b4d4a82703c331b1da20e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:02.273926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7253ee9d-5f4c-4959-b046-b1544d1839a1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:02.278648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 89/178] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:02 2022] GET /identity/v3/users/5c8b0f52ee8b4d4a82703c331b1da20e => generated 275 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:02.278648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1855e0-0648-45f6-aa35-d5e245e2411c None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.278648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1855e0-0648-45f6-aa35-d5e245e2411c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.278648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb1855e0-0648-45f6-aa35-d5e245e2411c None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.279520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 90/179] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:02.284807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1855e0-0648-45f6-aa35-d5e245e2411c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:02.390148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb4d83-c686-4493-8188-8b08cd6cd2ff None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:02.400017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb1855e0-0648-45f6-aa35-d5e245e2411c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:02.435765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d950b0-af1f-4a62-92c7-723cedebe81f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:02.436641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d950b0-af1f-4a62-92c7-723cedebe81f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.437006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d950b0-af1f-4a62-92c7-723cedebe81f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.437231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d950b0-af1f-4a62-92c7-723cedebe81f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.439301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8ab185-ec6a-4ff2-bb20-1b7c6be8bc80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:02.439950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ab185-ec6a-4ff2-bb20-1b7c6be8bc80 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.440233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ab185-ec6a-4ff2-bb20-1b7c6be8bc80 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.440513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8ab185-ec6a-4ff2-bb20-1b7c6be8bc80 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.441374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8ab185-ec6a-4ff2-bb20-1b7c6be8bc80 None admin] RBAC: Adding query filter params (role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569, user.id=5c8b0f52ee8b4d4a82703c331b1da20e) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:02.441686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8ab185-ec6a-4ff2-bb20-1b7c6be8bc80 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:02.443142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8ab185-ec6a-4ff2-bb20-1b7c6be8bc80 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:02.447615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d950b0-af1f-4a62-92c7-723cedebe81f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:02.449815 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 91/180] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:02 2022] GET /identity/v3/role_assignments?user.id=5c8b0f52ee8b4d4a82703c331b1da20e&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:02.453113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8ab185-ec6a-4ff2-bb20-1b7c6be8bc80 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-b2d950b0-af1f-4a62-92c7-723cedebe81f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:02.456186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:02.457280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 90/181] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:45:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:02.458982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a24991-0f77-4298-bfd7-d4476db0eed7 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.460204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a24991-0f77-4298-bfd7-d4476db0eed7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.460204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a24991-0f77-4298-bfd7-d4476db0eed7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.479535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-16a24991-0f77-4298-bfd7-d4476db0eed7 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:02.480871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-16a24991-0f77-4298-bfd7-d4476db0eed7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:02.493811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:02.534914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:02.557614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 91/182] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:02.559847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd97a71-2921-440a-9a1f-c1818f59ab0a None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.560139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd97a71-2921-440a-9a1f-c1818f59ab0a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.560419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd97a71-2921-440a-9a1f-c1818f59ab0a None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.572945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 92/183] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:02.572945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb4ee48-5f11-496d-9389-3df9f16e109e None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.572945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb4ee48-5f11-496d-9389-3df9f16e109e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.572945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb4ee48-5f11-496d-9389-3df9f16e109e None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.572945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 93/184] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:02.572945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb4ee48-5f11-496d-9389-3df9f16e109e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:02.573462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8ab185-ec6a-4ff2-bb20-1b7c6be8bc80 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:02.610145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8618812e-4738-4174-8ce7-915d542459de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:02.610813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8618812e-4738-4174-8ce7-915d542459de None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.611099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8618812e-4738-4174-8ce7-915d542459de None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.611378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8618812e-4738-4174-8ce7-915d542459de None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.687677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8618812e-4738-4174-8ce7-915d542459de None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:02.689572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8618812e-4738-4174-8ce7-915d542459de None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:02.699583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb4ee48-5f11-496d-9389-3df9f16e109e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:02.721928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:02.736059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a19db4e-2734-4218-9338-642d6baed4f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:02.736831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a19db4e-2734-4218-9338-642d6baed4f2 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.737118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a19db4e-2734-4218-9338-642d6baed4f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.737392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a19db4e-2734-4218-9338-642d6baed4f2 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.738139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a19db4e-2734-4218-9338-642d6baed4f2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:02.738445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a19db4e-2734-4218-9338-642d6baed4f2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:02.739826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a19db4e-2734-4218-9338-642d6baed4f2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:02.748270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 94/185] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:02.749525 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a204d02-0a61-4e92-82c1-c2197f726b28 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.749821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a204d02-0a61-4e92-82c1-c2197f726b28 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.750101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a204d02-0a61-4e92-82c1-c2197f726b28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.766467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-7a204d02-0a61-4e92-82c1-c2197f726b28 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:02.767638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7a204d02-0a61-4e92-82c1-c2197f726b28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:02.790869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 92/186] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 340 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:02.792700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9431c8c2-cee1-445a-99aa-74413b9fb332 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.793041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9431c8c2-cee1-445a-99aa-74413b9fb332 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.793326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9431c8c2-cee1-445a-99aa-74413b9fb332 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.802052 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:02.809205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9431c8c2-cee1-445a-99aa-74413b9fb332 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:02.810377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9431c8c2-cee1-445a-99aa-74413b9fb332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:02.832934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 95/187] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:02.833934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a204d02-0a61-4e92-82c1-c2197f726b28 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:02.881315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 93/188] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:02.882342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9431c8c2-cee1-445a-99aa-74413b9fb332 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:02.932496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a204d02-0a61-4e92-82c1-c2197f726b28 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:02.942783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:02.966662 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b22f56-65f5-4e7c-9f2c-0f1532cffeda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:02.967452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b22f56-65f5-4e7c-9f2c-0f1532cffeda None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:02.967727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b22f56-65f5-4e7c-9f2c-0f1532cffeda None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:02.967999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b22f56-65f5-4e7c-9f2c-0f1532cffeda None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:02.977975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78b22f56-65f5-4e7c-9f2c-0f1532cffeda None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:02.980280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9431c8c2-cee1-445a-99aa-74413b9fb332 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-78b22f56-65f5-4e7c-9f2c-0f1532cffeda None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:02.986652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:02.987632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:02.988594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 96/189] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:02 2022] GET /identity/v3/roles/reader => generated 84 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:02.989724 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b22f56-65f5-4e7c-9f2c-0f1532cffeda None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.013954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1487b6dc-677c-4675-a299-630552d403d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.014826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1487b6dc-677c-4675-a299-630552d403d6 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.015108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1487b6dc-677c-4675-a299-630552d403d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.015388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1487b6dc-677c-4675-a299-630552d403d6 None admin] PATH_INFO: `/v3/users/b35e4dfcc2d3481e905247e20c2e365d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.021134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:03.028255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1487b6dc-677c-4675-a299-630552d403d6 None admin] RBAC: Authorizing `identity:get_user(user_id=b35e4dfcc2d3481e905247e20c2e365d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:03.030010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1487b6dc-677c-4675-a299-630552d403d6 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:03.031869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 94/190] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:02 2022] GET /identity/v3/users/b35e4dfcc2d3481e905247e20c2e365d => generated 283 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:03.032437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1487b6dc-677c-4675-a299-630552d403d6 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.097901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78b22f56-65f5-4e7c-9f2c-0f1532cffeda None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.127978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1487b6dc-677c-4675-a299-630552d403d6 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.130418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f7d77-e213-4f5f-a45d-19916ba1e0e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.131269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f7d77-e213-4f5f-a45d-19916ba1e0e1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.131545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f7d77-e213-4f5f-a45d-19916ba1e0e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.131809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c2f7d77-e213-4f5f-a45d-19916ba1e0e1 None admin] PATH_INFO: `/v3/projects/edf1e3eb7289421a9782aa94730b2021` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.133002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2f7d77-e213-4f5f-a45d-19916ba1e0e1 None admin] RBAC: Authorizing `identity:get_project(project_id=edf1e3eb7289421a9782aa94730b2021)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:03.134460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c2f7d77-e213-4f5f-a45d-19916ba1e0e1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:03.136661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 97/191] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:02 2022] GET /identity/v3/projects/edf1e3eb7289421a9782aa94730b2021 => generated 312 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:03.137442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f7d77-e213-4f5f-a45d-19916ba1e0e1 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.154362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:03.167802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c00c440-a72a-4116-8b82-9bfeab9e6644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.168654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c00c440-a72a-4116-8b82-9bfeab9e6644 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.168960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c00c440-a72a-4116-8b82-9bfeab9e6644 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.169234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c00c440-a72a-4116-8b82-9bfeab9e6644 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.170384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c00c440-a72a-4116-8b82-9bfeab9e6644 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:03.171841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c00c440-a72a-4116-8b82-9bfeab9e6644 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:03.173569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 95/192] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:03.174357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c00c440-a72a-4116-8b82-9bfeab9e6644 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.232496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:03.235117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c2f7d77-e213-4f5f-a45d-19916ba1e0e1 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.269236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7312f170-9daf-4b66-b1a7-22725ea32cf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.270025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7312f170-9daf-4b66-b1a7-22725ea32cf5 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.270183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c00c440-a72a-4116-8b82-9bfeab9e6644 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.270303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7312f170-9daf-4b66-b1a7-22725ea32cf5 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.270573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7312f170-9daf-4b66-b1a7-22725ea32cf5 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.280539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7312f170-9daf-4b66-b1a7-22725ea32cf5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-7312f170-9daf-4b66-b1a7-22725ea32cf5 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:03.288460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:03.289386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:03.290167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 98/193] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:03.290846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7312f170-9daf-4b66-b1a7-22725ea32cf5 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.304700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bbd03a-4710-4772-94e4-52fe1c5a3385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.305389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bbd03a-4710-4772-94e4-52fe1c5a3385 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.305673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bbd03a-4710-4772-94e4-52fe1c5a3385 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.306161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bbd03a-4710-4772-94e4-52fe1c5a3385 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.306700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bbd03a-4710-4772-94e4-52fe1c5a3385 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:03.307010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bbd03a-4710-4772-94e4-52fe1c5a3385 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:03.308932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bbd03a-4710-4772-94e4-52fe1c5a3385 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:03.316844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 96/194] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:03 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:03.317619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bbd03a-4710-4772-94e4-52fe1c5a3385 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.370207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:03.396924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7312f170-9daf-4b66-b1a7-22725ea32cf5 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.419353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bbd03a-4710-4772-94e4-52fe1c5a3385 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.428637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b9d91-697c-40c5-88d9-5ff68a1ab634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.429310 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b9d91-697c-40c5-88d9-5ff68a1ab634 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.429588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b9d91-697c-40c5-88d9-5ff68a1ab634 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.429864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-436b9d91-697c-40c5-88d9-5ff68a1ab634 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.452856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb15465-8a77-49da-aa2c-430776b08905 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.453497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb15465-8a77-49da-aa2c-430776b08905 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.453777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb15465-8a77-49da-aa2c-430776b08905 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.454050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb15465-8a77-49da-aa2c-430776b08905 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.454869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb15465-8a77-49da-aa2c-430776b08905 None admin] RBAC: Adding query filter params (role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=edf1e3eb7289421a9782aa94730b2021, user.id=5c8b0f52ee8b4d4a82703c331b1da20e) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:03.455178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb15465-8a77-49da-aa2c-430776b08905 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:03.456616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb15465-8a77-49da-aa2c-430776b08905 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:03.457214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:03.462455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 97/195] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:03 2022] GET /identity/v3/role_assignments?user.id=5c8b0f52ee8b4d4a82703c331b1da20e&scope.project.id=edf1e3eb7289421a9782aa94730b2021&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:03.463306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb15465-8a77-49da-aa2c-430776b08905 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.523402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436b9d91-697c-40c5-88d9-5ff68a1ab634 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:03.524879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-436b9d91-697c-40c5-88d9-5ff68a1ab634 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:03.579141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb15465-8a77-49da-aa2c-430776b08905 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.596058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:03.621054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729c4d3d-c5c4-4e93-9cca-309b0ca10bfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.621757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729c4d3d-c5c4-4e93-9cca-309b0ca10bfb None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.622046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729c4d3d-c5c4-4e93-9cca-309b0ca10bfb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.622315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729c4d3d-c5c4-4e93-9cca-309b0ca10bfb None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.625281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729c4d3d-c5c4-4e93-9cca-309b0ca10bfb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:03.625281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729c4d3d-c5c4-4e93-9cca-309b0ca10bfb None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:03.639317 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 99/196] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:03.639554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b9d91-697c-40c5-88d9-5ff68a1ab634 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.695028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 98/197] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:45:03 2022] POST /identity/v3/users => generated 297 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:45:03.695940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729c4d3d-c5c4-4e93-9cca-309b0ca10bfb None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.709184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:03.768290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-436b9d91-697c-40c5-88d9-5ff68a1ab634 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.805186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c03602d-c726-486d-afd7-4c93f1dc2463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.806071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c03602d-c726-486d-afd7-4c93f1dc2463 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.806355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c03602d-c726-486d-afd7-4c93f1dc2463 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.806627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c03602d-c726-486d-afd7-4c93f1dc2463 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.807386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c03602d-c726-486d-afd7-4c93f1dc2463 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:03.807692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c03602d-c726-486d-afd7-4c93f1dc2463 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:03.809702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c03602d-c726-486d-afd7-4c93f1dc2463 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:03.819501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 100/198] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:03.824037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729c4d3d-c5c4-4e93-9cca-309b0ca10bfb None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.825138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c03602d-c726-486d-afd7-4c93f1dc2463 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.862714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90543689-f9a5-4d76-812e-67ed184c5c77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:03.863594 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90543689-f9a5-4d76-812e-67ed184c5c77 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:03.863881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90543689-f9a5-4d76-812e-67ed184c5c77 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:03.864159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90543689-f9a5-4d76-812e-67ed184c5c77 None admin] PATH_INFO: `/v3/users/aa2f0b25d6414893bb607fdcecb88286` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:03.865380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90543689-f9a5-4d76-812e-67ed184c5c77 None admin] RBAC: Authorizing `identity:get_user(user_id=aa2f0b25d6414893bb607fdcecb88286)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:03.866843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90543689-f9a5-4d76-812e-67ed184c5c77 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:03.868492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 99/199] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:03 2022] GET /identity/v3/users/aa2f0b25d6414893bb607fdcecb88286 => generated 289 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:03.869290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90543689-f9a5-4d76-812e-67ed184c5c77 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:03.916707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:03.952167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c03602d-c726-486d-afd7-4c93f1dc2463 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:03.984625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:03.998544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90543689-f9a5-4d76-812e-67ed184c5c77 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:04.004682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3677c52d-6822-4054-870e-a6582e9c9114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:04.005949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3677c52d-6822-4054-870e-a6582e9c9114 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:04.006410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3677c52d-6822-4054-870e-a6582e9c9114 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:04.006985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3677c52d-6822-4054-870e-a6582e9c9114 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:04.008488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3677c52d-6822-4054-870e-a6582e9c9114 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:04.010341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3677c52d-6822-4054-870e-a6582e9c9114 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:04.012962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 101/200] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:03 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:04.014005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3677c52d-6822-4054-870e-a6582e9c9114 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:04.047397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84773124-7e72-4e63-a4ed-2effebd02265 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:04.048283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84773124-7e72-4e63-a4ed-2effebd02265 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:04.048853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84773124-7e72-4e63-a4ed-2effebd02265 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:04.048853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84773124-7e72-4e63-a4ed-2effebd02265 None admin] PATH_INFO: `/v3/users/b35e4dfcc2d3481e905247e20c2e365d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:04.050883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84773124-7e72-4e63-a4ed-2effebd02265 None admin] RBAC: Authorizing `identity:get_user(user_id=b35e4dfcc2d3481e905247e20c2e365d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:04.052375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84773124-7e72-4e63-a4ed-2effebd02265 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:04.054343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 100/201] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:03 2022] GET /identity/v3/users/b35e4dfcc2d3481e905247e20c2e365d => generated 283 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:04.054912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84773124-7e72-4e63-a4ed-2effebd02265 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:04.135206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3677c52d-6822-4054-870e-a6582e9c9114 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:04.171144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5494de70-d011-4a4a-85ba-5db640efc871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:04.171794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5494de70-d011-4a4a-85ba-5db640efc871 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:04.172081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5494de70-d011-4a4a-85ba-5db640efc871 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:04.172359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5494de70-d011-4a4a-85ba-5db640efc871 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:04.194835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84773124-7e72-4e63-a4ed-2effebd02265 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:04.206106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:04.236177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:04.244843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49e91f8-f5cb-4f19-914c-7dfbf41e830a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:04.246577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49e91f8-f5cb-4f19-914c-7dfbf41e830a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:04.247029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49e91f8-f5cb-4f19-914c-7dfbf41e830a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:04.247156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5494de70-d011-4a4a-85ba-5db640efc871 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:04.247630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a49e91f8-f5cb-4f19-914c-7dfbf41e830a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:04.248567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5494de70-d011-4a4a-85ba-5db640efc871 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:04.249005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49e91f8-f5cb-4f19-914c-7dfbf41e830a None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=b77e955a802a489daa9bd13993979e40, user.id=b35e4dfcc2d3481e905247e20c2e365d) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:04.249514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49e91f8-f5cb-4f19-914c-7dfbf41e830a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:04.252165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a49e91f8-f5cb-4f19-914c-7dfbf41e830a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:04.265214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 101/202] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:04 2022] GET /identity/v3/role_assignments?user.id=b35e4dfcc2d3481e905247e20c2e365d&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=766799d1b6c049168590571f1a959b3b => generated 259 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:04.265214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49e91f8-f5cb-4f19-914c-7dfbf41e830a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:04.468682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a49e91f8-f5cb-4f19-914c-7dfbf41e830a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:04.471060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 102/203] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:04.476384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5494de70-d011-4a4a-85ba-5db640efc871 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:04.550954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac8405a-5427-4e1f-94ec-1595f4597e04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:04.551615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac8405a-5427-4e1f-94ec-1595f4597e04 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:04.551903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac8405a-5427-4e1f-94ec-1595f4597e04 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:04.552182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac8405a-5427-4e1f-94ec-1595f4597e04 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:04.589261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:04.602541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5494de70-d011-4a4a-85ba-5db640efc871 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:04.660561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350a0116-0fdb-44ae-95e5-0448f3474434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:04.661597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:04.663299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350a0116-0fdb-44ae-95e5-0448f3474434 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:04.663759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350a0116-0fdb-44ae-95e5-0448f3474434 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:04.664174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350a0116-0fdb-44ae-95e5-0448f3474434 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:04.665572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350a0116-0fdb-44ae-95e5-0448f3474434 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:04.668813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350a0116-0fdb-44ae-95e5-0448f3474434 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:04.673233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 103/204] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:04 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:04.678765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350a0116-0fdb-44ae-95e5-0448f3474434 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:04.726170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac8405a-5427-4e1f-94ec-1595f4597e04 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:04.728291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac8405a-5427-4e1f-94ec-1595f4597e04 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:04.864634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 102/205] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 603 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:04.870134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac8405a-5427-4e1f-94ec-1595f4597e04 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:04.888645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350a0116-0fdb-44ae-95e5-0448f3474434 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:04.953793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e426a1a6-2475-4b34-99b0-0493d49a03e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:04.954458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e426a1a6-2475-4b34-99b0-0493d49a03e2 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:04.954737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e426a1a6-2475-4b34-99b0-0493d49a03e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:04.955014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e426a1a6-2475-4b34-99b0-0493d49a03e2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:04.956644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e426a1a6-2475-4b34-99b0-0493d49a03e2 None admin] RBAC: Adding query filter params (role.id=94339d681b5c4fd188991c590c897ad4, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569, user.id=aa2f0b25d6414893bb607fdcecb88286) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:04.956985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e426a1a6-2475-4b34-99b0-0493d49a03e2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:04.958456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e426a1a6-2475-4b34-99b0-0493d49a03e2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:04.971485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 104/206] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:04 2022] GET /identity/v3/role_assignments?user.id=aa2f0b25d6414893bb607fdcecb88286&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=94339d681b5c4fd188991c590c897ad4 => generated 608 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:04.988106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:05.077946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac8405a-5427-4e1f-94ec-1595f4597e04 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:05.125144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9438a1c-b8df-403b-b4b7-fe03dbc1ffe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:05.126030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9438a1c-b8df-403b-b4b7-fe03dbc1ffe7 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:05.126316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9438a1c-b8df-403b-b4b7-fe03dbc1ffe7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:05.126598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9438a1c-b8df-403b-b4b7-fe03dbc1ffe7 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:05.128651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9438a1c-b8df-403b-b4b7-fe03dbc1ffe7 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:05.130192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9438a1c-b8df-403b-b4b7-fe03dbc1ffe7 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:05.132265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 103/207] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:04 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 263 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:05.137840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e426a1a6-2475-4b34-99b0-0493d49a03e2 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:05.169337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:05.252894 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e426a1a6-2475-4b34-99b0-0493d49a03e2 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:05.295027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c50de8d-65fa-471a-a836-9cef7cf9e4f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:05.295691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c50de8d-65fa-471a-a836-9cef7cf9e4f0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:05.295977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c50de8d-65fa-471a-a836-9cef7cf9e4f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:05.296257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c50de8d-65fa-471a-a836-9cef7cf9e4f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:05.297117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c50de8d-65fa-471a-a836-9cef7cf9e4f0 None admin] RBAC: Adding query filter params (role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=b77e955a802a489daa9bd13993979e40, user.id=b35e4dfcc2d3481e905247e20c2e365d) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:05.297430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c50de8d-65fa-471a-a836-9cef7cf9e4f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:05.298894 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c50de8d-65fa-471a-a836-9cef7cf9e4f0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:05.304884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 105/208] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:05 2022] GET /identity/v3/role_assignments?user.id=b35e4dfcc2d3481e905247e20c2e365d&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 259 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:06.179695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3d32a1-6cb9-44ed-b2d2-41b2bdc4afd1 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.180217 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3d32a1-6cb9-44ed-b2d2-41b2bdc4afd1 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.180545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3d32a1-6cb9-44ed-b2d2-41b2bdc4afd1 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.182358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 104/209] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:06.187992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7e7816-7f28-47eb-bc7f-97b9411dfd55 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.188413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7e7816-7f28-47eb-bc7f-97b9411dfd55 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.188734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7e7816-7f28-47eb-bc7f-97b9411dfd55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.206318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-af7e7816-7f28-47eb-bc7f-97b9411dfd55 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:06.206612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c866012-221c-4a32-94b5-f48d54ea9f75 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.206824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c866012-221c-4a32-94b5-f48d54ea9f75 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.207107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c866012-221c-4a32-94b5-f48d54ea9f75 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.207514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-af7e7816-7f28-47eb-bc7f-97b9411dfd55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:06.208606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 105/210] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:06.213524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f23ea86-bdd9-4cf5-bb65-3f6779a3fc0a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.213817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f23ea86-bdd9-4cf5-bb65-3f6779a3fc0a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.214166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f23ea86-bdd9-4cf5-bb65-3f6779a3fc0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.232515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-6f23ea86-bdd9-4cf5-bb65-3f6779a3fc0a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:06.233845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6f23ea86-bdd9-4cf5-bb65-3f6779a3fc0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:06.291861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 106/211] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:06.306946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 106/212] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:06.316333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7e7816-7f28-47eb-bc7f-97b9411dfd55 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:06.333821 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f23ea86-bdd9-4cf5-bb65-3f6779a3fc0a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:06.345741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:06.423319 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:06.447464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af7e7816-7f28-47eb-bc7f-97b9411dfd55 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:06.464643 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f23ea86-bdd9-4cf5-bb65-3f6779a3fc0a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:06.483326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2dbcb-8699-4091-97a9-5ee375562ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:06.484157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2dbcb-8699-4091-97a9-5ee375562ece None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.484442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2dbcb-8699-4091-97a9-5ee375562ece None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.484716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2dbcb-8699-4091-97a9-5ee375562ece None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.497531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d2dbcb-8699-4091-97a9-5ee375562ece None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:06.500503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332131d4-bac1-4e7f-a96f-e1cd5539437a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:06.501375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332131d4-bac1-4e7f-a96f-e1cd5539437a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.501659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332131d4-bac1-4e7f-a96f-e1cd5539437a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.501935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-332131d4-bac1-4e7f-a96f-e1cd5539437a None admin] PATH_INFO: `/v3/domains/default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.503233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332131d4-bac1-4e7f-a96f-e1cd5539437a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:06.504699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-332131d4-bac1-4e7f-a96f-e1cd5539437a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-35d2dbcb-8699-4091-97a9-5ee375562ece None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:06.505579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:06.506432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:06.507205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:06.507795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 107/213] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:06 2022] GET /identity/v3/domains/default => generated 199 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:06.507861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 107/214] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:06.511720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332131d4-bac1-4e7f-a96f-e1cd5539437a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:06.516030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2dbcb-8699-4091-97a9-5ee375562ece None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:06.621727 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-332131d4-bac1-4e7f-a96f-e1cd5539437a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:06.625194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:06.653261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2dbcb-8699-4091-97a9-5ee375562ece None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:06.658470 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa2a76b-1245-4ff0-b8aa-a8de814d9a7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:06.659140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa2a76b-1245-4ff0-b8aa-a8de814d9a7b None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.659421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa2a76b-1245-4ff0-b8aa-a8de814d9a7b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.659696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa2a76b-1245-4ff0-b8aa-a8de814d9a7b None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.660411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa2a76b-1245-4ff0-b8aa-a8de814d9a7b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:06.661988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa2a76b-1245-4ff0-b8aa-a8de814d9a7b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:06.671049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:06.690155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21df1741-991e-4a32-899e-b425c8dfce56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:06.690817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df1741-991e-4a32-899e-b425c8dfce56 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.691100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df1741-991e-4a32-899e-b425c8dfce56 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.691380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df1741-991e-4a32-899e-b425c8dfce56 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.692131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df1741-991e-4a32-899e-b425c8dfce56 None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:06.692440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df1741-991e-4a32-899e-b425c8dfce56 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:06.693905 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df1741-991e-4a32-899e-b425c8dfce56 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:06.702113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 108/215] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:06 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:06.704404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cc8f6e-58f1-4075-9e8f-b71b89161a02 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.704742 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cc8f6e-58f1-4075-9e8f-b71b89161a02 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.705057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24cc8f6e-58f1-4075-9e8f-b71b89161a02 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.706397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 109/216] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:06.707475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cc8f6e-58f1-4075-9e8f-b71b89161a02 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:06.752171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 108/217] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:45:06 2022] POST /identity/v3/users => generated 297 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:45:06.753837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4606c01e-848c-47d4-a26c-762e69d62530 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.754335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4606c01e-848c-47d4-a26c-762e69d62530 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.754678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4606c01e-848c-47d4-a26c-762e69d62530 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.778661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-4606c01e-848c-47d4-a26c-762e69d62530 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:06.780241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4606c01e-848c-47d4-a26c-762e69d62530 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:06.844533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24cc8f6e-58f1-4075-9e8f-b71b89161a02 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:06.866999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 109/218] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:06.889032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88028932-dbda-4ef0-aab6-87e55bcbeb06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:06.890073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88028932-dbda-4ef0-aab6-87e55bcbeb06 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:06.890356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88028932-dbda-4ef0-aab6-87e55bcbeb06 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:06.890654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88028932-dbda-4ef0-aab6-87e55bcbeb06 None admin] PATH_INFO: `/v3/users/3f12d54c24824da3ae16de1d9198dad0` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:06.891026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4606c01e-848c-47d4-a26c-762e69d62530 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:06.899014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:06.907328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88028932-dbda-4ef0-aab6-87e55bcbeb06 None admin] RBAC: Authorizing `identity:get_user(user_id=3f12d54c24824da3ae16de1d9198dad0)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:06.908961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88028932-dbda-4ef0-aab6-87e55bcbeb06 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:06.911457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 110/219] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:06 2022] GET /identity/v3/users/3f12d54c24824da3ae16de1d9198dad0 => generated 297 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:06.916451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88028932-dbda-4ef0-aab6-87e55bcbeb06 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:07.030859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4606c01e-848c-47d4-a26c-762e69d62530 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:07.040031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88028932-dbda-4ef0-aab6-87e55bcbeb06 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:07.062807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:07.078743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eaa95f-84c1-417b-91d4-b79d2419929e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:07.079865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eaa95f-84c1-417b-91d4-b79d2419929e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.080439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eaa95f-84c1-417b-91d4-b79d2419929e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.080899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3eaa95f-84c1-417b-91d4-b79d2419929e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.081556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646021dc-8de6-4355-9982-007585cc222d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:07.082401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646021dc-8de6-4355-9982-007585cc222d None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.082775 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646021dc-8de6-4355-9982-007585cc222d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.083120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646021dc-8de6-4355-9982-007585cc222d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.096488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3eaa95f-84c1-417b-91d4-b79d2419929e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-c3eaa95f-84c1-417b-91d4-b79d2419929e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:07.111384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:07.112334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:07.113168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 110/220] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:45:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:07.119375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eaa95f-84c1-417b-91d4-b79d2419929e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:07.175579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:07.183823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646021dc-8de6-4355-9982-007585cc222d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:07.185549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-646021dc-8de6-4355-9982-007585cc222d None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:07.258225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3eaa95f-84c1-417b-91d4-b79d2419929e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:07.305548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77782ae2-444b-45e2-9dc9-75d6d36ae7cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:07.306134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77782ae2-444b-45e2-9dc9-75d6d36ae7cf None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.306419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77782ae2-444b-45e2-9dc9-75d6d36ae7cf None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.306697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77782ae2-444b-45e2-9dc9-75d6d36ae7cf None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.307456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77782ae2-444b-45e2-9dc9-75d6d36ae7cf None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:07.308299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77782ae2-444b-45e2-9dc9-75d6d36ae7cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:07.310414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77782ae2-444b-45e2-9dc9-75d6d36ae7cf None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:07.326788 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 111/221] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:07.333559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77782ae2-444b-45e2-9dc9-75d6d36ae7cf None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:07.334156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 111/222] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jun 10 12:45:07.343233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646021dc-8de6-4355-9982-007585cc222d None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:07.440375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:07.540420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77782ae2-444b-45e2-9dc9-75d6d36ae7cf None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:07.545346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:07.551895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646021dc-8de6-4355-9982-007585cc222d None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:07.587077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d110e2f6-891f-4955-81f1-043fc4bba5d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:07.588141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d110e2f6-891f-4955-81f1-043fc4bba5d2 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.588429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d110e2f6-891f-4955-81f1-043fc4bba5d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.588704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d110e2f6-891f-4955-81f1-043fc4bba5d2 None admin] PATH_INFO: `/v3/users/b35e4dfcc2d3481e905247e20c2e365d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.590019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d110e2f6-891f-4955-81f1-043fc4bba5d2 None admin] RBAC: Authorizing `identity:get_user(user_id=b35e4dfcc2d3481e905247e20c2e365d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:07.591488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d110e2f6-891f-4955-81f1-043fc4bba5d2 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:07.592637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fe85be-c226-49c1-9270-225084d6f176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:07.593488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 112/223] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:07 2022] GET /identity/v3/users/b35e4dfcc2d3481e905247e20c2e365d => generated 283 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:07.593941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fe85be-c226-49c1-9270-225084d6f176 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.594427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fe85be-c226-49c1-9270-225084d6f176 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.594769 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d05df25-d451-471a-bb62-d5396ecc80a3 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.595059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d05df25-d451-471a-bb62-d5396ecc80a3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.595278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fe85be-c226-49c1-9270-225084d6f176 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.595372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d05df25-d451-471a-bb62-d5396ecc80a3 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.596815 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 113/224] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:07.597037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fe85be-c226-49c1-9270-225084d6f176 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:07.598562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fe85be-c226-49c1-9270-225084d6f176 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:07.600324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 112/225] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:07 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:07.602379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef8d4ad-4f80-4735-a3ac-f106fcc04cb8 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.602826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef8d4ad-4f80-4735-a3ac-f106fcc04cb8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.603099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fe85be-c226-49c1-9270-225084d6f176 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:07.603372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef8d4ad-4f80-4735-a3ac-f106fcc04cb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.625637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-0ef8d4ad-4f80-4735-a3ac-f106fcc04cb8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:07.627910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0ef8d4ad-4f80-4735-a3ac-f106fcc04cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:07.718124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fe85be-c226-49c1-9270-225084d6f176 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:07.720144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 114/226] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:07.721201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef8d4ad-4f80-4735-a3ac-f106fcc04cb8 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:07.770737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:07.775893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2187300-9547-4f8b-80c8-96bf4df4aca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:07.776795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2187300-9547-4f8b-80c8-96bf4df4aca4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.777105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2187300-9547-4f8b-80c8-96bf4df4aca4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.777444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2187300-9547-4f8b-80c8-96bf4df4aca4 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.778631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2187300-9547-4f8b-80c8-96bf4df4aca4 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:07.780427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2187300-9547-4f8b-80c8-96bf4df4aca4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:07.783736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 113/227] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:07 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:07.787810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2187300-9547-4f8b-80c8-96bf4df4aca4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:07.821353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:07.833978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef8d4ad-4f80-4735-a3ac-f106fcc04cb8 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:07.872198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad37d1fb-b00f-466c-9e2c-4a3e4ba559b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:07.873125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad37d1fb-b00f-466c-9e2c-4a3e4ba559b6 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.873600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad37d1fb-b00f-466c-9e2c-4a3e4ba559b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.874019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad37d1fb-b00f-466c-9e2c-4a3e4ba559b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.875100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad37d1fb-b00f-466c-9e2c-4a3e4ba559b6 None admin] RBAC: Adding query filter params (role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=b77e955a802a489daa9bd13993979e40, user.id=3f12d54c24824da3ae16de1d9198dad0) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:07.875602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad37d1fb-b00f-466c-9e2c-4a3e4ba559b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:07.877249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad37d1fb-b00f-466c-9e2c-4a3e4ba559b6 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:07.883900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 115/228] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:07 2022] GET /identity/v3/role_assignments?user.id=3f12d54c24824da3ae16de1d9198dad0&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:07.884998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad37d1fb-b00f-466c-9e2c-4a3e4ba559b6 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:07.895054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2187300-9547-4f8b-80c8-96bf4df4aca4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:07.937802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc768ed-1860-478c-9843-83883b4b8923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:07.938639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc768ed-1860-478c-9843-83883b4b8923 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:07.938895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc768ed-1860-478c-9843-83883b4b8923 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:07.939170 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc768ed-1860-478c-9843-83883b4b8923 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:07.950293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc768ed-1860-478c-9843-83883b4b8923 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-6cc768ed-1860-478c-9843-83883b4b8923 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:07.958947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:07.969877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 114/229] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:07.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc768ed-1860-478c-9843-83883b4b8923 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:08.133450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad37d1fb-b00f-466c-9e2c-4a3e4ba559b6 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:08.145131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:08.165559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:08.184787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc768ed-1860-478c-9843-83883b4b8923 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:08.189899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f677ad2-dbce-4789-9f85-d80a953b48ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:08.190811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f677ad2-dbce-4789-9f85-d80a953b48ff None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:08.191094 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f677ad2-dbce-4789-9f85-d80a953b48ff None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:08.191372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f677ad2-dbce-4789-9f85-d80a953b48ff None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40/users/b35e4dfcc2d3481e905247e20c2e365d/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:08.193436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f677ad2-dbce-4789-9f85-d80a953b48ff None admin] RBAC: Authorizing `identity:create_grant(project_id=b77e955a802a489daa9bd13993979e40, user_id=b35e4dfcc2d3481e905247e20c2e365d, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:08.195008 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f677ad2-dbce-4789-9f85-d80a953b48ff None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:08.219966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9423bbeb-1064-4e68-af70-ddafee483f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:08.220746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9423bbeb-1064-4e68-af70-ddafee483f2c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:08.221070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9423bbeb-1064-4e68-af70-ddafee483f2c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:08.221456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9423bbeb-1064-4e68-af70-ddafee483f2c None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:08.222254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9423bbeb-1064-4e68-af70-ddafee483f2c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:08.222572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9423bbeb-1064-4e68-af70-ddafee483f2c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:08.224051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9423bbeb-1064-4e68-af70-ddafee483f2c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:08.237958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 115/230] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:08.239378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9423bbeb-1064-4e68-af70-ddafee483f2c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:08.281659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 116/231] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:45:07 2022] PUT /identity/v3/projects/b77e955a802a489daa9bd13993979e40/users/b35e4dfcc2d3481e905247e20c2e365d/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:08.360376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9423bbeb-1064-4e68-af70-ddafee483f2c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:08.400129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea628b4-71cd-46d1-9a13-415780a776e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:08.401029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea628b4-71cd-46d1-9a13-415780a776e0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:08.401355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea628b4-71cd-46d1-9a13-415780a776e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:08.401640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea628b4-71cd-46d1-9a13-415780a776e0 None admin] PATH_INFO: `/v3/users/b35e4dfcc2d3481e905247e20c2e365d` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:08.402859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea628b4-71cd-46d1-9a13-415780a776e0 None admin] RBAC: Authorizing `identity:get_user(user_id=b35e4dfcc2d3481e905247e20c2e365d)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:08.404575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea628b4-71cd-46d1-9a13-415780a776e0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:08.406475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 116/232] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:08 2022] GET /identity/v3/users/b35e4dfcc2d3481e905247e20c2e365d => generated 283 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:08.412268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f677ad2-dbce-4789-9f85-d80a953b48ff None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:08.569737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f677ad2-dbce-4789-9f85-d80a953b48ff None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:08.606057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3ac97d-3df5-4af4-90eb-fba12adeaedb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:08.606973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3ac97d-3df5-4af4-90eb-fba12adeaedb None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:08.607257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3ac97d-3df5-4af4-90eb-fba12adeaedb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:08.607528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3ac97d-3df5-4af4-90eb-fba12adeaedb None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:08.608707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3ac97d-3df5-4af4-90eb-fba12adeaedb None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:08.610257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3ac97d-3df5-4af4-90eb-fba12adeaedb None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:08.611969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 117/233] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:08 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:08.617917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea628b4-71cd-46d1-9a13-415780a776e0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:08.646230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:08.735829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea628b4-71cd-46d1-9a13-415780a776e0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:08.777360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa1fee9-98f4-402d-bb69-d3e00b1f5962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:08.778578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa1fee9-98f4-402d-bb69-d3e00b1f5962 None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:08.779300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa1fee9-98f4-402d-bb69-d3e00b1f5962 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:08.779746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa1fee9-98f4-402d-bb69-d3e00b1f5962 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40/users/b35e4dfcc2d3481e905247e20c2e365d/roles/df56ac8b9c6f46958342cc16f10cea63` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:08.782282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa1fee9-98f4-402d-bb69-d3e00b1f5962 None admin] RBAC: Authorizing `identity:create_grant(project_id=b77e955a802a489daa9bd13993979e40, user_id=b35e4dfcc2d3481e905247e20c2e365d, role_id=df56ac8b9c6f46958342cc16f10cea63)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:08.783947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa1fee9-98f4-402d-bb69-d3e00b1f5962 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:08.854299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 117/234] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:45:08 2022] PUT /identity/v3/projects/b77e955a802a489daa9bd13993979e40/users/b35e4dfcc2d3481e905247e20c2e365d/roles/df56ac8b9c6f46958342cc16f10cea63 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:09.439545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b76c34-aca6-455d-aa26-27a75698b45d None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:09.440046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b76c34-aca6-455d-aa26-27a75698b45d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:09.440147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b76c34-aca6-455d-aa26-27a75698b45d None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:09.441664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 118/235] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:09.446450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151195d2-3684-4ef5-918e-8b0dcaa94b75 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:09.446832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151195d2-3684-4ef5-918e-8b0dcaa94b75 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:09.447002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151195d2-3684-4ef5-918e-8b0dcaa94b75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:09.472326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ee785-2d0b-4d20-a03a-2146ca3a44a7 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:09.472618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ee785-2d0b-4d20-a03a-2146ca3a44a7 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:09.472905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ee785-2d0b-4d20-a03a-2146ca3a44a7 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:09.474506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 119/236] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:09.475405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-151195d2-3684-4ef5-918e-8b0dcaa94b75 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:09.476590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-151195d2-3684-4ef5-918e-8b0dcaa94b75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:09.479230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337407bc-88a6-4a94-9c2f-549ade89a3ba None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:09.480666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337407bc-88a6-4a94-9c2f-549ade89a3ba None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:09.481245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337407bc-88a6-4a94-9c2f-549ade89a3ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:09.503851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-337407bc-88a6-4a94-9c2f-549ade89a3ba None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:09.505063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-337407bc-88a6-4a94-9c2f-549ade89a3ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:09.537468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:09.574390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 120/237] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:09.594703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 118/238] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:09.608512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337407bc-88a6-4a94-9c2f-549ade89a3ba None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:09.620069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151195d2-3684-4ef5-918e-8b0dcaa94b75 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:09.640044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:09.727110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337407bc-88a6-4a94-9c2f-549ade89a3ba None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:09.772462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b779c-0bfe-4bf7-9269-ac8ba75b644c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:09.773672 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b779c-0bfe-4bf7-9269-ac8ba75b644c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:09.774154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b779c-0bfe-4bf7-9269-ac8ba75b644c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:09.774654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457b779c-0bfe-4bf7-9269-ac8ba75b644c None admin] PATH_INFO: `/v3/domains/default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:09.776310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457b779c-0bfe-4bf7-9269-ac8ba75b644c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:09.779591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-457b779c-0bfe-4bf7-9269-ac8ba75b644c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:09.782772 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 121/239] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:09.787845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b779c-0bfe-4bf7-9269-ac8ba75b644c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:09.815777 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151195d2-3684-4ef5-918e-8b0dcaa94b75 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:09.853806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feaf8b0-44e9-4c09-9a8b-f56b2ed1a01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:09.855751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feaf8b0-44e9-4c09-9a8b-f56b2ed1a01d None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:09.856234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feaf8b0-44e9-4c09-9a8b-f56b2ed1a01d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:09.856798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feaf8b0-44e9-4c09-9a8b-f56b2ed1a01d None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:09.869009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5feaf8b0-44e9-4c09-9a8b-f56b2ed1a01d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-5feaf8b0-44e9-4c09-9a8b-f56b2ed1a01d None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:09.878967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:09.879925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:09.880715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:09.883957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 119/240] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:09 2022] GET /identity/v3/roles/reader => generated 84 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:09.885135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:09.890133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feaf8b0-44e9-4c09-9a8b-f56b2ed1a01d None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:09.965199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:09.996567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feaf8b0-44e9-4c09-9a8b-f56b2ed1a01d None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:10.023802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457b779c-0bfe-4bf7-9269-ac8ba75b644c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:10.034733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8673c181-21ac-46f6-838f-64c5db475deb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:10.035742 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8673c181-21ac-46f6-838f-64c5db475deb None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:10.036245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8673c181-21ac-46f6-838f-64c5db475deb None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:10.036744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8673c181-21ac-46f6-838f-64c5db475deb None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:10.037730 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8673c181-21ac-46f6-838f-64c5db475deb None admin] RBAC: Adding query filter params (name=reader) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:10.038288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8673c181-21ac-46f6-838f-64c5db475deb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:10.040030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8673c181-21ac-46f6-838f-64c5db475deb None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:10.049372 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 120/241] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:09 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:10.054342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8673c181-21ac-46f6-838f-64c5db475deb None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:10.113798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4247f47-1411-4009-9d55-f32604006890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:10.114136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4247f47-1411-4009-9d55-f32604006890 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:10.114423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4247f47-1411-4009-9d55-f32604006890 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:10.114700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4247f47-1411-4009-9d55-f32604006890 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:10.115423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4247f47-1411-4009-9d55-f32604006890 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:10.116894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4247f47-1411-4009-9d55-f32604006890 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:10.167037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8673c181-21ac-46f6-838f-64c5db475deb None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:10.203060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cd2a92-8e82-4848-ad4a-c42fc09402da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:10.204341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cd2a92-8e82-4848-ad4a-c42fc09402da None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:10.204871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cd2a92-8e82-4848-ad4a-c42fc09402da None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:10.205365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6cd2a92-8e82-4848-ad4a-c42fc09402da None admin] PATH_INFO: `/v3/users/f471a18fc01a4fb2a2df6a3f86749926` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:10.209058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:10.220674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cd2a92-8e82-4848-ad4a-c42fc09402da None admin] RBAC: Authorizing `identity:get_user(user_id=f471a18fc01a4fb2a2df6a3f86749926)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:10.222765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6cd2a92-8e82-4848-ad4a-c42fc09402da None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:10.225046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 121/242] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:10 2022] GET /identity/v3/users/f471a18fc01a4fb2a2df6a3f86749926 => generated 297 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:10.233016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cd2a92-8e82-4848-ad4a-c42fc09402da None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:10.271636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 122/243] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:45:09 2022] POST /identity/v3/users => generated 293 bytes in 484 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:45:10.377137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6cd2a92-8e82-4848-ad4a-c42fc09402da None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:10.435996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade96266-4e91-45ce-b53d-f34346d5964c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:10.436668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade96266-4e91-45ce-b53d-f34346d5964c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:10.436974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade96266-4e91-45ce-b53d-f34346d5964c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:10.437251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade96266-4e91-45ce-b53d-f34346d5964c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:10.440788 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5272472-5c21-4752-9ecd-536ebe473567 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:10.440788 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5272472-5c21-4752-9ecd-536ebe473567 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:10.440788 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5272472-5c21-4752-9ecd-536ebe473567 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:10.442354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 123/244] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:10.454246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc21e251-2a62-454e-9f0f-92384368acb2 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:10.454593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc21e251-2a62-454e-9f0f-92384368acb2 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:10.454800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc21e251-2a62-454e-9f0f-92384368acb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:10.487329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bc21e251-2a62-454e-9f0f-92384368acb2 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:10.492375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bc21e251-2a62-454e-9f0f-92384368acb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:10.540525 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:10.551487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade96266-4e91-45ce-b53d-f34346d5964c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:10.553235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade96266-4e91-45ce-b53d-f34346d5964c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:10.581840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 124/245] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:10.611505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc21e251-2a62-454e-9f0f-92384368acb2 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:10.632779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:10.695901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 122/246] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jun 10 12:45:10.702189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade96266-4e91-45ce-b53d-f34346d5964c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:10.739915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc21e251-2a62-454e-9f0f-92384368acb2 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:10.794629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9feebdb0-a23c-487a-8c22-d85883399cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:10.795445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9feebdb0-a23c-487a-8c22-d85883399cb4 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:10.795725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9feebdb0-a23c-487a-8c22-d85883399cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:10.796001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9feebdb0-a23c-487a-8c22-d85883399cb4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:10.810516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9feebdb0-a23c-487a-8c22-d85883399cb4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-9feebdb0-a23c-487a-8c22-d85883399cb4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:10.821314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:10.822257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:10.823219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:10.826465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 125/247] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:10 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:10.830744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:10.830957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9feebdb0-a23c-487a-8c22-d85883399cb4 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:10.844754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade96266-4e91-45ce-b53d-f34346d5964c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:10.890871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ad0ecc-ae6e-432c-90dd-951aaaf23b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:10.891670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ad0ecc-ae6e-432c-90dd-951aaaf23b46 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:10.892028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ad0ecc-ae6e-432c-90dd-951aaaf23b46 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:10.892540 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ad0ecc-ae6e-432c-90dd-951aaaf23b46 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:10.894225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ad0ecc-ae6e-432c-90dd-951aaaf23b46 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:10.895709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ad0ecc-ae6e-432c-90dd-951aaaf23b46 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:10.898334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 123/248] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:10 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:10.905242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ad0ecc-ae6e-432c-90dd-951aaaf23b46 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:10.936591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:10.963559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9feebdb0-a23c-487a-8c22-d85883399cb4 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:11.022106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f632e4-acdd-40c6-b0a4-1ad49e55c1ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:11.023167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f632e4-acdd-40c6-b0a4-1ad49e55c1ca None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.023649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f632e4-acdd-40c6-b0a4-1ad49e55c1ca None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.024122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f632e4-acdd-40c6-b0a4-1ad49e55c1ca None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.025113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f632e4-acdd-40c6-b0a4-1ad49e55c1ca None admin] RBAC: Adding query filter params (name=member) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:11.025637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f632e4-acdd-40c6-b0a4-1ad49e55c1ca None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:11.027800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49f632e4-acdd-40c6-b0a4-1ad49e55c1ca None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:11.032374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ad0ecc-ae6e-432c-90dd-951aaaf23b46 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:11.047279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 126/249] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:11.070301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f632e4-acdd-40c6-b0a4-1ad49e55c1ca None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:11.074921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de16ac58-8750-4313-a59d-67bfb5e47403 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:11.075627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de16ac58-8750-4313-a59d-67bfb5e47403 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.075918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de16ac58-8750-4313-a59d-67bfb5e47403 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.076197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de16ac58-8750-4313-a59d-67bfb5e47403 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.077078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de16ac58-8750-4313-a59d-67bfb5e47403 None admin] RBAC: Adding query filter params (role.id=94339d681b5c4fd188991c590c897ad4, scope.project.id=b77e955a802a489daa9bd13993979e40, user.id=f471a18fc01a4fb2a2df6a3f86749926) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:11.077392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de16ac58-8750-4313-a59d-67bfb5e47403 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:11.079104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de16ac58-8750-4313-a59d-67bfb5e47403 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:11.089431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 124/250] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:10 2022] GET /identity/v3/role_assignments?user.id=f471a18fc01a4fb2a2df6a3f86749926&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=94339d681b5c4fd188991c590c897ad4 => generated 259 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:11.230125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb17b6a-80c0-4e29-9623-c51727771bb5 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.230452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb17b6a-80c0-4e29-9623-c51727771bb5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.230697 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb17b6a-80c0-4e29-9623-c51727771bb5 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.232102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 125/251] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:45:11.237435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd7ae11-565c-46ad-bbc9-69fd257d33c6 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.237723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd7ae11-565c-46ad-bbc9-69fd257d33c6 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.237999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd7ae11-565c-46ad-bbc9-69fd257d33c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.241603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f632e4-acdd-40c6-b0a4-1ad49e55c1ca None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:11.270105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ccd7ae11-565c-46ad-bbc9-69fd257d33c6 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:11.271521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ccd7ae11-565c-46ad-bbc9-69fd257d33c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:11.297949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:11.348358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b78ee63-cc86-475d-a1ed-d35af0770c28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:11.349307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b78ee63-cc86-475d-a1ed-d35af0770c28 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.349586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b78ee63-cc86-475d-a1ed-d35af0770c28 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.349859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b78ee63-cc86-475d-a1ed-d35af0770c28 None admin] PATH_INFO: `/v3/users/3f12d54c24824da3ae16de1d9198dad0` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.353647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b78ee63-cc86-475d-a1ed-d35af0770c28 None admin] RBAC: Authorizing `identity:get_user(user_id=3f12d54c24824da3ae16de1d9198dad0)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:11.355768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b78ee63-cc86-475d-a1ed-d35af0770c28 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:11.357796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 127/252] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:11 2022] GET /identity/v3/users/3f12d54c24824da3ae16de1d9198dad0 => generated 297 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:11.364933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b78ee63-cc86-475d-a1ed-d35af0770c28 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:11.381836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:11.387242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 126/253] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:11.456478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd7ae11-565c-46ad-bbc9-69fd257d33c6 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:11.548295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b78ee63-cc86-475d-a1ed-d35af0770c28 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:11.592310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5396948c-dcb9-472c-9e6d-227dcdc3482a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:11.593437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5396948c-dcb9-472c-9e6d-227dcdc3482a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.593859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5396948c-dcb9-472c-9e6d-227dcdc3482a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.594273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5396948c-dcb9-472c-9e6d-227dcdc3482a None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.598084 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5396948c-dcb9-472c-9e6d-227dcdc3482a None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:11.600783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5396948c-dcb9-472c-9e6d-227dcdc3482a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:11.602884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 128/254] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:11 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:11.605113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a483c-cb45-4f24-9e02-f26d72d9482a None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.605712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a483c-cb45-4f24-9e02-f26d72d9482a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.607653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a483c-cb45-4f24-9e02-f26d72d9482a None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.610082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 129/255] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:11 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:11.611710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a483c-cb45-4f24-9e02-f26d72d9482a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:11.641806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd7ae11-565c-46ad-bbc9-69fd257d33c6 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:11.703166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d88c89-a5a3-40ec-884f-7e28adb92174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:11.704198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d88c89-a5a3-40ec-884f-7e28adb92174 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.704644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d88c89-a5a3-40ec-884f-7e28adb92174 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.705077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d88c89-a5a3-40ec-884f-7e28adb92174 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.716601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d88c89-a5a3-40ec-884f-7e28adb92174 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-85d88c89-a5a3-40ec-884f-7e28adb92174 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:11.725560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:11.726624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:11.727571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:11.730497 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 127/256] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:45:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:11.731951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a6c2df-8f18-4a9a-94b0-017111aae0e5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.732370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a6c2df-8f18-4a9a-94b0-017111aae0e5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.732821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a6c2df-8f18-4a9a-94b0-017111aae0e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.751547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a3a6c2df-8f18-4a9a-94b0-017111aae0e5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:11.752994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a3a6c2df-8f18-4a9a-94b0-017111aae0e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:11.771313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:11.772221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:11.831761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a483c-cb45-4f24-9e02-f26d72d9482a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:11.869153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbba73d-ebbe-4581-a4c3-7a16f9f75808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:11.870581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbba73d-ebbe-4581-a4c3-7a16f9f75808 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:11.871073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbba73d-ebbe-4581-a4c3-7a16f9f75808 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:11.871551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbba73d-ebbe-4581-a4c3-7a16f9f75808 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40/users/3f12d54c24824da3ae16de1d9198dad0/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:11.873849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbba73d-ebbe-4581-a4c3-7a16f9f75808 None admin] RBAC: Authorizing `identity:create_grant(project_id=b77e955a802a489daa9bd13993979e40, user_id=3f12d54c24824da3ae16de1d9198dad0, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:11.875528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbba73d-ebbe-4581-a4c3-7a16f9f75808 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:11.887960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 128/257] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:11.888661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a6c2df-8f18-4a9a-94b0-017111aae0e5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:11.945530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 130/258] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:45:11 2022] PUT /identity/v3/projects/b77e955a802a489daa9bd13993979e40/users/3f12d54c24824da3ae16de1d9198dad0/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:11.946929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbba73d-ebbe-4581-a4c3-7a16f9f75808 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:12.103125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a6c2df-8f18-4a9a-94b0-017111aae0e5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:12.159332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbba73d-ebbe-4581-a4c3-7a16f9f75808 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:12.168851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f3e1bd-1b14-42be-8d2a-06f9d6588b52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:12.169965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f3e1bd-1b14-42be-8d2a-06f9d6588b52 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:12.170448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f3e1bd-1b14-42be-8d2a-06f9d6588b52 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:12.170924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f3e1bd-1b14-42be-8d2a-06f9d6588b52 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:12.171996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f3e1bd-1b14-42be-8d2a-06f9d6588b52 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:12.172524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f3e1bd-1b14-42be-8d2a-06f9d6588b52 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:12.174323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f3e1bd-1b14-42be-8d2a-06f9d6588b52 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:12.191928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 129/259] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:11 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:12.197851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f3e1bd-1b14-42be-8d2a-06f9d6588b52 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:12.212049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:12.222699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ee2fae-cd21-4720-92fb-a8c3640fd30a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:12.223671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ee2fae-cd21-4720-92fb-a8c3640fd30a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:12.223959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ee2fae-cd21-4720-92fb-a8c3640fd30a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:12.224239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ee2fae-cd21-4720-92fb-a8c3640fd30a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:12.235389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:12.243085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ee2fae-cd21-4720-92fb-a8c3640fd30a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-86ee2fae-cd21-4720-92fb-a8c3640fd30a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:12.252064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:12.253075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:12.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 131/260] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:11 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:12.263441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ee2fae-cd21-4720-92fb-a8c3640fd30a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:12.396501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f3e1bd-1b14-42be-8d2a-06f9d6588b52 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:12.434959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ee2fae-cd21-4720-92fb-a8c3640fd30a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:12.449771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92aa572-fcc8-4a63-badc-41955be870c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:12.450687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92aa572-fcc8-4a63-badc-41955be870c3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:12.450974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92aa572-fcc8-4a63-badc-41955be870c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:12.451361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92aa572-fcc8-4a63-badc-41955be870c3 None admin] PATH_INFO: `/v3/users/b35e4dfcc2d3481e905247e20c2e365d` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:12.452693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92aa572-fcc8-4a63-badc-41955be870c3 None admin] RBAC: Authorizing `identity:get_user(user_id=b35e4dfcc2d3481e905247e20c2e365d)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:12.454320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92aa572-fcc8-4a63-badc-41955be870c3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:12.456428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 130/261] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:12 2022] GET /identity/v3/users/b35e4dfcc2d3481e905247e20c2e365d => generated 283 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:12.461817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92aa572-fcc8-4a63-badc-41955be870c3 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:12.497973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c99adc8-fb69-47b4-be07-bf6c38c0723b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:12.498702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c99adc8-fb69-47b4-be07-bf6c38c0723b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:12.498991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c99adc8-fb69-47b4-be07-bf6c38c0723b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:12.499270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c99adc8-fb69-47b4-be07-bf6c38c0723b None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:12.500036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c99adc8-fb69-47b4-be07-bf6c38c0723b None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:12.500341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c99adc8-fb69-47b4-be07-bf6c38c0723b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:12.501774 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c99adc8-fb69-47b4-be07-bf6c38c0723b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:12.516792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 132/262] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:12.522187 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c99adc8-fb69-47b4-be07-bf6c38c0723b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:12.577179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:12.615314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92aa572-fcc8-4a63-badc-41955be870c3 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:12.638278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:12.681615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bacc380-2973-4b4e-9171-cfe39426acbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:12.682333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bacc380-2973-4b4e-9171-cfe39426acbb None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:12.682562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bacc380-2973-4b4e-9171-cfe39426acbb None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:12.682847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bacc380-2973-4b4e-9171-cfe39426acbb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:12.707029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c99adc8-fb69-47b4-be07-bf6c38c0723b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:12.742844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed7820b-f07c-4ee8-bdd6-09bea0998609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:12.743740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed7820b-f07c-4ee8-bdd6-09bea0998609 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:12.744026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed7820b-f07c-4ee8-bdd6-09bea0998609 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:12.744306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed7820b-f07c-4ee8-bdd6-09bea0998609 None admin] PATH_INFO: `/v3/users/b35e4dfcc2d3481e905247e20c2e365d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:12.745630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed7820b-f07c-4ee8-bdd6-09bea0998609 None admin] RBAC: Authorizing `identity:get_user(user_id=b35e4dfcc2d3481e905247e20c2e365d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:12.747100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed7820b-f07c-4ee8-bdd6-09bea0998609 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:12.748745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 133/263] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:12 2022] GET /identity/v3/users/b35e4dfcc2d3481e905247e20c2e365d => generated 283 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:12.754311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed7820b-f07c-4ee8-bdd6-09bea0998609 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:12.790543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bacc380-2973-4b4e-9171-cfe39426acbb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:12.792556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bacc380-2973-4b4e-9171-cfe39426acbb None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:12.858975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed7820b-f07c-4ee8-bdd6-09bea0998609 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:12.894406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f256b1a1-ffbe-4db5-b032-931ace735414 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:12.895034 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f256b1a1-ffbe-4db5-b032-931ace735414 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:12.895327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f256b1a1-ffbe-4db5-b032-931ace735414 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:12.895609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f256b1a1-ffbe-4db5-b032-931ace735414 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:12.901268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:12.946457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:12.969704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f256b1a1-ffbe-4db5-b032-931ace735414 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:12.971667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f256b1a1-ffbe-4db5-b032-931ace735414 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:13.043843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 131/264] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 582 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:13.052787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bacc380-2973-4b4e-9171-cfe39426acbb None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:13.176636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 134/265] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jun 10 12:45:13.180791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bacc380-2973-4b4e-9171-cfe39426acbb None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:13.183425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f256b1a1-ffbe-4db5-b032-931ace735414 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:13.229765 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:13.238448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a85aa7-ccdc-4b10-bb53-72ae396a0d60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:13.239562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a85aa7-ccdc-4b10-bb53-72ae396a0d60 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:13.240015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a85aa7-ccdc-4b10-bb53-72ae396a0d60 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:13.240459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a85aa7-ccdc-4b10-bb53-72ae396a0d60 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:13.243600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a85aa7-ccdc-4b10-bb53-72ae396a0d60 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:13.245319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a85aa7-ccdc-4b10-bb53-72ae396a0d60 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:13.247434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 132/266] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:13 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:13.260268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a85aa7-ccdc-4b10-bb53-72ae396a0d60 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:13.340407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f256b1a1-ffbe-4db5-b032-931ace735414 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:13.348455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:13.405122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bd23f1-2897-44e1-8099-45d930be0cf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:13.405991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bd23f1-2897-44e1-8099-45d930be0cf1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:13.406272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bd23f1-2897-44e1-8099-45d930be0cf1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:13.406551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bd23f1-2897-44e1-8099-45d930be0cf1 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:13.410966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bd23f1-2897-44e1-8099-45d930be0cf1 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:13.412561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bd23f1-2897-44e1-8099-45d930be0cf1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:13.414732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 135/267] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:13 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:13.415977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69c85e7-56d6-42ba-8768-6431fcc85eb8 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:13.416316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69c85e7-56d6-42ba-8768-6431fcc85eb8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:13.417145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69c85e7-56d6-42ba-8768-6431fcc85eb8 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:13.418734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 136/268] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:13.424016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457f473-c283-4389-95d1-e2018f90b2ca None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:13.424306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457f473-c283-4389-95d1-e2018f90b2ca None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:13.424587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457f473-c283-4389-95d1-e2018f90b2ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:13.450826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a85aa7-ccdc-4b10-bb53-72ae396a0d60 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:13.454091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3457f473-c283-4389-95d1-e2018f90b2ca None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:13.460007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3457f473-c283-4389-95d1-e2018f90b2ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:13.522322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4eada-26dd-4c5b-88e3-6d861f536533 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:13.524756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4eada-26dd-4c5b-88e3-6d861f536533 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:13.527201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4eada-26dd-4c5b-88e3-6d861f536533 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:13.529539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b4eada-26dd-4c5b-88e3-6d861f536533 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:13.530802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4eada-26dd-4c5b-88e3-6d861f536533 None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=b77e955a802a489daa9bd13993979e40, user.id=b35e4dfcc2d3481e905247e20c2e365d) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:13.532339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4eada-26dd-4c5b-88e3-6d861f536533 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:13.534674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59b4eada-26dd-4c5b-88e3-6d861f536533 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:13.554161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 133/269] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:13 2022] GET /identity/v3/role_assignments?user.id=b35e4dfcc2d3481e905247e20c2e365d&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=766799d1b6c049168590571f1a959b3b => generated 608 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:13.555816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4eada-26dd-4c5b-88e3-6d861f536533 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:13.605554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 137/270] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:13.642284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457f473-c283-4389-95d1-e2018f90b2ca None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:13.693454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:13.818836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b4eada-26dd-4c5b-88e3-6d861f536533 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:13.826519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:13.867708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457f473-c283-4389-95d1-e2018f90b2ca None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:13.869125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d7f0b9-8f05-4ec2-a9e3-40f39723e4f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:13.870731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7f0b9-8f05-4ec2-a9e3-40f39723e4f7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:13.871219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7f0b9-8f05-4ec2-a9e3-40f39723e4f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:13.871687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d7f0b9-8f05-4ec2-a9e3-40f39723e4f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:13.872712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d7f0b9-8f05-4ec2-a9e3-40f39723e4f7 None admin] RBAC: Adding query filter params (role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=b77e955a802a489daa9bd13993979e40, user.id=b35e4dfcc2d3481e905247e20c2e365d) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:13.873535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d7f0b9-8f05-4ec2-a9e3-40f39723e4f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:13.876153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4d7f0b9-8f05-4ec2-a9e3-40f39723e4f7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:13.883023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 134/271] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:13 2022] GET /identity/v3/role_assignments?user.id=b35e4dfcc2d3481e905247e20c2e365d&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 608 bytes in 328 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:13.921723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d98203-d278-430c-a297-17b0d8a7ad66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:13.922948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d98203-d278-430c-a297-17b0d8a7ad66 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:13.923426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d98203-d278-430c-a297-17b0d8a7ad66 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:13.923917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d98203-d278-430c-a297-17b0d8a7ad66 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:13.939840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d98203-d278-430c-a297-17b0d8a7ad66 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-84d98203-d278-430c-a297-17b0d8a7ad66 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:13.947932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:13.949461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:13.950257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 138/272] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:13.957328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d7f0b9-8f05-4ec2-a9e3-40f39723e4f7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:14.029589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78485e1-49e0-4a6c-a7f3-0368bd38f8bf None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.029928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78485e1-49e0-4a6c-a7f3-0368bd38f8bf None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.030161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78485e1-49e0-4a6c-a7f3-0368bd38f8bf None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.031929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 139/273] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:45:14.037710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b668cc-b758-4716-8426-5a5630df53a3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.037993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b668cc-b758-4716-8426-5a5630df53a3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.038281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b668cc-b758-4716-8426-5a5630df53a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.070575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-01b668cc-b758-4716-8426-5a5630df53a3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:14.071726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-01b668cc-b758-4716-8426-5a5630df53a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:14.137077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4d7f0b9-8f05-4ec2-a9e3-40f39723e4f7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:14.189326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 140/274] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:14.197547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349d72a2-c318-462f-a627-0e1ed70259a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:14.199054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349d72a2-c318-462f-a627-0e1ed70259a4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.199512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349d72a2-c318-462f-a627-0e1ed70259a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.199979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-349d72a2-c318-462f-a627-0e1ed70259a4 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.200945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349d72a2-c318-462f-a627-0e1ed70259a4 None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:14.201425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349d72a2-c318-462f-a627-0e1ed70259a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:14.202987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-349d72a2-c318-462f-a627-0e1ed70259a4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:14.215030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 135/275] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:14.218027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b668cc-b758-4716-8426-5a5630df53a3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:14.220234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349d72a2-c318-462f-a627-0e1ed70259a4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:14.238530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:14.266783 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:14.355980 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b668cc-b758-4716-8426-5a5630df53a3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:14.373853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-349d72a2-c318-462f-a627-0e1ed70259a4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:14.399799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5a6f74-b6f8-45dd-9ffd-5e21e1b1786c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:14.400695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5a6f74-b6f8-45dd-9ffd-5e21e1b1786c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.402506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5a6f74-b6f8-45dd-9ffd-5e21e1b1786c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.402506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5a6f74-b6f8-45dd-9ffd-5e21e1b1786c None admin] PATH_INFO: `/v3/domains/default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.402977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5a6f74-b6f8-45dd-9ffd-5e21e1b1786c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:14.404963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5a6f74-b6f8-45dd-9ffd-5e21e1b1786c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:14.408892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 141/276] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:14.416564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5a6f74-b6f8-45dd-9ffd-5e21e1b1786c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:14.417936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d620fc2-8ff6-44b7-a927-7b289b0d74a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:14.419061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d620fc2-8ff6-44b7-a927-7b289b0d74a4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.419534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d620fc2-8ff6-44b7-a927-7b289b0d74a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.421259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d620fc2-8ff6-44b7-a927-7b289b0d74a4 None admin] PATH_INFO: `/v3/users/1f755d0a272845638e015f44d42de665` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.437352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d620fc2-8ff6-44b7-a927-7b289b0d74a4 None admin] RBAC: Authorizing `identity:get_user(user_id=1f755d0a272845638e015f44d42de665)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:14.439183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d620fc2-8ff6-44b7-a927-7b289b0d74a4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:14.441185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 136/277] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:14 2022] GET /identity/v3/users/1f755d0a272845638e015f44d42de665 => generated 293 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:14.442753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4928adb-77b4-46cf-918e-7f0dc38c5496 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.443149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4928adb-77b4-46cf-918e-7f0dc38c5496 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.443426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4928adb-77b4-46cf-918e-7f0dc38c5496 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.444937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 137/278] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:14.450092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4928adb-77b4-46cf-918e-7f0dc38c5496 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:14.527999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:14.551638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5a6f74-b6f8-45dd-9ffd-5e21e1b1786c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:14.562529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:14.575813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4928adb-77b4-46cf-918e-7f0dc38c5496 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:14.590155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f0bf5-e1b9-43d6-a424-d2fbf4a02746 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:14.590674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f0bf5-e1b9-43d6-a424-d2fbf4a02746 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.590958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f0bf5-e1b9-43d6-a424-d2fbf4a02746 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.591233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68f0bf5-e1b9-43d6-a424-d2fbf4a02746 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.591972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68f0bf5-e1b9-43d6-a424-d2fbf4a02746 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:14.593542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68f0bf5-e1b9-43d6-a424-d2fbf4a02746 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:14.613503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7360d6-31ce-4d29-bce0-e33f61622192 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:14.614164 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7360d6-31ce-4d29-bce0-e33f61622192 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.614455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7360d6-31ce-4d29-bce0-e33f61622192 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.614733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7360d6-31ce-4d29-bce0-e33f61622192 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.615566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7360d6-31ce-4d29-bce0-e33f61622192 None admin] RBAC: Adding query filter params (role.id=dfc4731db42b4f0bbb5e10d84814f9cd, user.id=1f755d0a272845638e015f44d42de665, scope.system=all) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:14.620878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7360d6-31ce-4d29-bce0-e33f61622192 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:14.620878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7360d6-31ce-4d29-bce0-e33f61622192 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:14.624082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 138/279] 10.222.0.25 () {56 vars in 1308 bytes} [Fri Jun 10 12:45:14 2022] GET /identity/v3/role_assignments?user.id=1f755d0a272845638e015f44d42de665&scope.system=all&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 226 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:14.625409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c3c8b5-2316-48cf-b0d4-1f1a187bd755 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.625707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c3c8b5-2316-48cf-b0d4-1f1a187bd755 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.625994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c3c8b5-2316-48cf-b0d4-1f1a187bd755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.651970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a8c3c8b5-2316-48cf-b0d4-1f1a187bd755 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:14.653272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a8c3c8b5-2316-48cf-b0d4-1f1a187bd755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:14.685549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 142/280] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:45:14 2022] POST /identity/v3/users => generated 293 bytes in 265 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:45:14.759502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 139/281] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:14.787962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f0bf5-e1b9-43d6-a424-d2fbf4a02746 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:14.935209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68f0bf5-e1b9-43d6-a424-d2fbf4a02746 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:14.955106 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ef992d-4045-4766-aa4a-3b06126d8e93 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.955604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ef992d-4045-4766-aa4a-3b06126d8e93 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.955973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ef992d-4045-4766-aa4a-3b06126d8e93 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.958060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 140/282] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:14.965146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e49cd1e-1872-42fb-9073-cd35f0d8572c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.965565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e49cd1e-1872-42fb-9073-cd35f0d8572c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.965985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e49cd1e-1872-42fb-9073-cd35f0d8572c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.981604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc13a19-cb5e-4d27-88f2-192fd734f8ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:14.983162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc13a19-cb5e-4d27-88f2-192fd734f8ae None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:14.983454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc13a19-cb5e-4d27-88f2-192fd734f8ae None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:14.983737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc13a19-cb5e-4d27-88f2-192fd734f8ae None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:14.989051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6e49cd1e-1872-42fb-9073-cd35f0d8572c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:14.990322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6e49cd1e-1872-42fb-9073-cd35f0d8572c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:14.993297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:14.999264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc13a19-cb5e-4d27-88f2-192fd734f8ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-1bc13a19-cb5e-4d27-88f2-192fd734f8ae None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:15.007100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:15.011701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:15.012512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 143/283] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:14 2022] GET /identity/v3/roles/reader => generated 84 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:15.017158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc13a19-cb5e-4d27-88f2-192fd734f8ae None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:15.132801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 141/284] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:15.160325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e49cd1e-1872-42fb-9073-cd35f0d8572c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:15.162463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc13a19-cb5e-4d27-88f2-192fd734f8ae None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:15.211486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eca5cbd-b9c4-42ac-97f1-a6d9f879831f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:15.212697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca5cbd-b9c4-42ac-97f1-a6d9f879831f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:15.214207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca5cbd-b9c4-42ac-97f1-a6d9f879831f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:15.214727 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eca5cbd-b9c4-42ac-97f1-a6d9f879831f None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:15.215987 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eca5cbd-b9c4-42ac-97f1-a6d9f879831f None admin] RBAC: Adding query filter params (name=reader) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:15.216658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eca5cbd-b9c4-42ac-97f1-a6d9f879831f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:15.218609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eca5cbd-b9c4-42ac-97f1-a6d9f879831f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:15.232208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 144/285] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:15 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:15.233671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:15.237493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eca5cbd-b9c4-42ac-97f1-a6d9f879831f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:15.287696 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e49cd1e-1872-42fb-9073-cd35f0d8572c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:15.337939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:15.364335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd173a-dc03-4543-9e95-4a110642b708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:15.365335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd173a-dc03-4543-9e95-4a110642b708 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:15.365723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd173a-dc03-4543-9e95-4a110642b708 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:15.366102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd173a-dc03-4543-9e95-4a110642b708 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:15.387487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cd173a-dc03-4543-9e95-4a110642b708 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c2cd173a-dc03-4543-9e95-4a110642b708 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:15.401754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:15.402828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:15.404072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:15.407421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 142/286] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:15.414365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd173a-dc03-4543-9e95-4a110642b708 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:15.424915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eca5cbd-b9c4-42ac-97f1-a6d9f879831f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:15.495120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a66b5d9-82db-4e66-8d6f-4922b27542df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:15.495927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a66b5d9-82db-4e66-8d6f-4922b27542df None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:15.496213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a66b5d9-82db-4e66-8d6f-4922b27542df None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:15.496495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a66b5d9-82db-4e66-8d6f-4922b27542df None admin] PATH_INFO: `/v3/users/f471a18fc01a4fb2a2df6a3f86749926` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:15.499759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a66b5d9-82db-4e66-8d6f-4922b27542df None admin] RBAC: Authorizing `identity:get_user(user_id=f471a18fc01a4fb2a2df6a3f86749926)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:15.504066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a66b5d9-82db-4e66-8d6f-4922b27542df None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:15.505757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 145/287] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:15 2022] GET /identity/v3/users/f471a18fc01a4fb2a2df6a3f86749926 => generated 297 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:15.511736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a66b5d9-82db-4e66-8d6f-4922b27542df None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:15.550849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:15.580473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd173a-dc03-4543-9e95-4a110642b708 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:15.618880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d3cbab-4770-4512-b746-2ed57311f4cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:15.619563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d3cbab-4770-4512-b746-2ed57311f4cf None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:15.620571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d3cbab-4770-4512-b746-2ed57311f4cf None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:15.620885 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d3cbab-4770-4512-b746-2ed57311f4cf None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:15.621740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d3cbab-4770-4512-b746-2ed57311f4cf None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:15.622051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d3cbab-4770-4512-b746-2ed57311f4cf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:15.623527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d3cbab-4770-4512-b746-2ed57311f4cf None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:15.633268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 143/288] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:15.639196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d3cbab-4770-4512-b746-2ed57311f4cf None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:15.701332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a66b5d9-82db-4e66-8d6f-4922b27542df None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:15.764930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:15.790099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a2ac6d-c453-4a79-8216-c40989b36ff3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:15.790829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d3cbab-4770-4512-b746-2ed57311f4cf None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:15.791008 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a2ac6d-c453-4a79-8216-c40989b36ff3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:15.791267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a2ac6d-c453-4a79-8216-c40989b36ff3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:15.792150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a2ac6d-c453-4a79-8216-c40989b36ff3 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:15.797569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a2ac6d-c453-4a79-8216-c40989b36ff3 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:15.799060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a2ac6d-c453-4a79-8216-c40989b36ff3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:15.801169 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 146/289] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:15 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 290 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:15.807295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a2ac6d-c453-4a79-8216-c40989b36ff3 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:15.850569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b38f6e6-1296-43f3-9ca9-cb6680d4342b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:15.852187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b38f6e6-1296-43f3-9ca9-cb6680d4342b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:15.852487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b38f6e6-1296-43f3-9ca9-cb6680d4342b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:15.852789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b38f6e6-1296-43f3-9ca9-cb6680d4342b None admin] PATH_INFO: `/v3/users/3f12d54c24824da3ae16de1d9198dad0` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:15.856017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b38f6e6-1296-43f3-9ca9-cb6680d4342b None admin] RBAC: Authorizing `identity:get_user(user_id=3f12d54c24824da3ae16de1d9198dad0)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:15.858727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b38f6e6-1296-43f3-9ca9-cb6680d4342b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:15.860408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 144/290] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:15 2022] GET /identity/v3/users/3f12d54c24824da3ae16de1d9198dad0 => generated 297 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:15.866194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b38f6e6-1296-43f3-9ca9-cb6680d4342b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:15.874043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:15.972810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a2ac6d-c453-4a79-8216-c40989b36ff3 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:16.008366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c4b367-da62-41ac-92e6-60e408cbbea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:16.009310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c4b367-da62-41ac-92e6-60e408cbbea2 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:16.009602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c4b367-da62-41ac-92e6-60e408cbbea2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:16.009886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60c4b367-da62-41ac-92e6-60e408cbbea2 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40/users/f471a18fc01a4fb2a2df6a3f86749926/roles/94339d681b5c4fd188991c590c897ad4` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:16.010915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b38f6e6-1296-43f3-9ca9-cb6680d4342b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:16.011984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c4b367-da62-41ac-92e6-60e408cbbea2 None admin] RBAC: Authorizing `identity:create_grant(project_id=b77e955a802a489daa9bd13993979e40, user_id=f471a18fc01a4fb2a2df6a3f86749926, role_id=94339d681b5c4fd188991c590c897ad4)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:16.013549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60c4b367-da62-41ac-92e6-60e408cbbea2 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:16.047994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea3eec7-e1a3-49c9-911b-4601f1e74dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:16.048976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea3eec7-e1a3-49c9-911b-4601f1e74dae None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:16.049460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea3eec7-e1a3-49c9-911b-4601f1e74dae None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:16.049954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea3eec7-e1a3-49c9-911b-4601f1e74dae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:16.098814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 147/291] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:45:15 2022] PUT /identity/v3/projects/b77e955a802a489daa9bd13993979e40/users/f471a18fc01a4fb2a2df6a3f86749926/roles/94339d681b5c4fd188991c590c897ad4 => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 12:45:16.123256 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea3eec7-e1a3-49c9-911b-4601f1e74dae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:16.125100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea3eec7-e1a3-49c9-911b-4601f1e74dae None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:16.128111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:16.252050 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 145/292] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:15 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 386 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:16.257702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c4b367-da62-41ac-92e6-60e408cbbea2 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:16.368887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:16.429828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60c4b367-da62-41ac-92e6-60e408cbbea2 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:16.465286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce362922-b604-4a02-94b1-18e63f8f595a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:16.466154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce362922-b604-4a02-94b1-18e63f8f595a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:16.466440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce362922-b604-4a02-94b1-18e63f8f595a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:16.466722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce362922-b604-4a02-94b1-18e63f8f595a None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:16.467920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce362922-b604-4a02-94b1-18e63f8f595a None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:16.471113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce362922-b604-4a02-94b1-18e63f8f595a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:16.472926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 148/293] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:16 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:16.479506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea3eec7-e1a3-49c9-911b-4601f1e74dae None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:16.608600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea3eec7-e1a3-49c9-911b-4601f1e74dae None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:16.617785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:16.664242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-867530f5-504d-4ff5-804b-245ee31258e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:16.664938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867530f5-504d-4ff5-804b-245ee31258e5 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:16.665232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867530f5-504d-4ff5-804b-245ee31258e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:16.665512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-867530f5-504d-4ff5-804b-245ee31258e5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:16.666352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867530f5-504d-4ff5-804b-245ee31258e5 None admin] RBAC: Adding query filter params (role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=b77e955a802a489daa9bd13993979e40, user.id=3f12d54c24824da3ae16de1d9198dad0) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:16.667187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867530f5-504d-4ff5-804b-245ee31258e5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:16.668644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-867530f5-504d-4ff5-804b-245ee31258e5 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:16.677441 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 146/294] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:16 2022] GET /identity/v3/role_assignments?user.id=3f12d54c24824da3ae16de1d9198dad0&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 608 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:17.138582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560a5021-7c23-4428-8c2f-c9738bd667df None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.139090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560a5021-7c23-4428-8c2f-c9738bd667df None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.139163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560a5021-7c23-4428-8c2f-c9738bd667df None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.140737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 149/295] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:17.147053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1ead86-8418-4801-806e-644b9ab02364 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.147489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1ead86-8418-4801-806e-644b9ab02364 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.147671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1ead86-8418-4801-806e-644b9ab02364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.166684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-0f1ead86-8418-4801-806e-644b9ab02364 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:17.167889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0f1ead86-8418-4801-806e-644b9ab02364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:17.249780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 147/296] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:17.281464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560a5021-7c23-4428-8c2f-c9738bd667df None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:17.413260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560a5021-7c23-4428-8c2f-c9738bd667df None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:17.439071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:17.451023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f6bc22-2e8e-4332-a242-5c34383b2f96 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.451320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f6bc22-2e8e-4332-a242-5c34383b2f96 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.451596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f6bc22-2e8e-4332-a242-5c34383b2f96 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.453186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 148/297] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:45:17.453890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e8da0c-2e81-4cb6-95be-f5540874362d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:17.454683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e8da0c-2e81-4cb6-95be-f5540874362d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.459560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e8da0c-2e81-4cb6-95be-f5540874362d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.459560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e8da0c-2e81-4cb6-95be-f5540874362d None admin] PATH_INFO: `/v3/roles/member` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.460018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f39d861-fde8-49b9-b110-0b668e25d49d None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.460309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f39d861-fde8-49b9-b110-0b668e25d49d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.460590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f39d861-fde8-49b9-b110-0b668e25d49d None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.461941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 149/298] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:17.462958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c9cb0-56b8-4188-a86d-bb0679a1c217 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.463262 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c9cb0-56b8-4188-a86d-bb0679a1c217 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.463573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c9cb0-56b8-4188-a86d-bb0679a1c217 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.468394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e8da0c-2e81-4cb6-95be-f5540874362d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-73e8da0c-2e81-4cb6-95be-f5540874362d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:17.476093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:17.477007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:17.477795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 150/299] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:17 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:17.478955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b903996-2c85-4d91-a64a-74ae8fe0889a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.479162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b903996-2c85-4d91-a64a-74ae8fe0889a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.480868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b903996-2c85-4d91-a64a-74ae8fe0889a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.482805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-9b3c9cb0-56b8-4188-a86d-bb0679a1c217 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:17.484019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9b3c9cb0-56b8-4188-a86d-bb0679a1c217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:17.496484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-0b903996-2c85-4d91-a64a-74ae8fe0889a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:17.497692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0b903996-2c85-4d91-a64a-74ae8fe0889a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:17.554556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 150/300] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:17.555845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3c9cb0-56b8-4188-a86d-bb0679a1c217 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:17.573963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 151/301] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:17.577651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b903996-2c85-4d91-a64a-74ae8fe0889a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:17.591120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:17.666576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3c9cb0-56b8-4188-a86d-bb0679a1c217 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:17.689804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b903996-2c85-4d91-a64a-74ae8fe0889a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:17.701374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4375c251-82a3-4937-9720-a3061faf41ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:17.702064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375c251-82a3-4937-9720-a3061faf41ec None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.702338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375c251-82a3-4937-9720-a3061faf41ec None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.702618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4375c251-82a3-4937-9720-a3061faf41ec None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.703385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4375c251-82a3-4937-9720-a3061faf41ec None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:17.703693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4375c251-82a3-4937-9720-a3061faf41ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:17.704939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:17.705116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4375c251-82a3-4937-9720-a3061faf41ec None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:17.718706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 151/302] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:17.718706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4375c251-82a3-4937-9720-a3061faf41ec None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:17.730569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ceb33-e1b1-470c-b7b7-a1c870689710 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:17.731366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ceb33-e1b1-470c-b7b7-a1c870689710 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.731658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ceb33-e1b1-470c-b7b7-a1c870689710 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.732031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727ceb33-e1b1-470c-b7b7-a1c870689710 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.733134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727ceb33-e1b1-470c-b7b7-a1c870689710 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:17.734597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727ceb33-e1b1-470c-b7b7-a1c870689710 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:17.736381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 152/303] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:17.741241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ceb33-e1b1-470c-b7b7-a1c870689710 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:17.818179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:17.829817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4375c251-82a3-4937-9720-a3061faf41ec None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:17.850313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727ceb33-e1b1-470c-b7b7-a1c870689710 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:17.864666 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06e4f6e-141e-45a5-a268-392fc9769d58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:17.866106 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06e4f6e-141e-45a5-a268-392fc9769d58 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.866409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06e4f6e-141e-45a5-a268-392fc9769d58 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.866689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06e4f6e-141e-45a5-a268-392fc9769d58 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.879254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06e4f6e-141e-45a5-a268-392fc9769d58 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:17.883931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ac5dae-336f-4657-82c1-cff695c926bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:17.884814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ac5dae-336f-4657-82c1-cff695c926bc None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:17.885606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ac5dae-336f-4657-82c1-cff695c926bc None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:17.885894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ac5dae-336f-4657-82c1-cff695c926bc None admin] PATH_INFO: `/v3/users/1f755d0a272845638e015f44d42de665` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:17.887074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ac5dae-336f-4657-82c1-cff695c926bc None admin] RBAC: Authorizing `identity:get_user(user_id=1f755d0a272845638e015f44d42de665)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-e06e4f6e-141e-45a5-a268-392fc9769d58 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:17.888036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:17.889008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:17.894988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:17.895612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ac5dae-336f-4657-82c1-cff695c926bc None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:17.895691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 152/304] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:17 2022] GET /identity/v3/roles/reader => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:17.895691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06e4f6e-141e-45a5-a268-392fc9769d58 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:17.895810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 153/305] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:17 2022] GET /identity/v3/users/1f755d0a272845638e015f44d42de665 => generated 293 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:17.895810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ac5dae-336f-4657-82c1-cff695c926bc None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:17.948698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:18.001588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ac5dae-336f-4657-82c1-cff695c926bc None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:18.003078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06e4f6e-141e-45a5-a268-392fc9769d58 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:18.035804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45f0439-e641-441a-bae6-ac8469f0cde4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:18.036507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45f0439-e641-441a-bae6-ac8469f0cde4 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:18.036838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45f0439-e641-441a-bae6-ac8469f0cde4 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:18.037184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d45f0439-e641-441a-bae6-ac8469f0cde4 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:18.037861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45f0439-e641-441a-bae6-ac8469f0cde4 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:18.038167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45f0439-e641-441a-bae6-ac8469f0cde4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:18.038668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92762e8-7cbe-4e4c-931b-6f43c76e4e2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:18.039329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92762e8-7cbe-4e4c-931b-6f43c76e4e2a None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:18.039537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d45f0439-e641-441a-bae6-ac8469f0cde4 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:18.039630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92762e8-7cbe-4e4c-931b-6f43c76e4e2a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:18.039873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92762e8-7cbe-4e4c-931b-6f43c76e4e2a None admin] PATH_INFO: `/v3/groups` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:18.040552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92762e8-7cbe-4e4c-931b-6f43c76e4e2a None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:18.042574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92762e8-7cbe-4e4c-931b-6f43c76e4e2a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:18.048955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 154/306] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:17 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:18.049752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45f0439-e641-441a-bae6-ac8469f0cde4 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:18.124800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 153/307] 10.222.0.25 () {60 vars in 1134 bytes} [Fri Jun 10 12:45:17 2022] POST /identity/v3/groups => generated 229 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:45:18.124800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92762e8-7cbe-4e4c-931b-6f43c76e4e2a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:18.136986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:18.169281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d45f0439-e641-441a-bae6-ac8469f0cde4 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:18.192606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:18.226320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22f08ee-4878-41f2-9df0-eb9595dcd0ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:18.228080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22f08ee-4878-41f2-9df0-eb9595dcd0ed None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:18.228915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22f08ee-4878-41f2-9df0-eb9595dcd0ed None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:18.229519 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22f08ee-4878-41f2-9df0-eb9595dcd0ed None admin] PATH_INFO: `/v3/system/users/1f755d0a272845638e015f44d42de665/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:18.231442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22f08ee-4878-41f2-9df0-eb9595dcd0ed None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=1f755d0a272845638e015f44d42de665, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:18.234016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22f08ee-4878-41f2-9df0-eb9595dcd0ed None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:18.254560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92762e8-7cbe-4e4c-931b-6f43c76e4e2a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:18.293464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca11529-0da8-459f-9c4a-9530a68e91e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:18.295122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca11529-0da8-459f-9c4a-9530a68e91e0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:18.295122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca11529-0da8-459f-9c4a-9530a68e91e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:18.295122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca11529-0da8-459f-9c4a-9530a68e91e0 None admin] PATH_INFO: `/v3/users/5866c6e8a00d4826b8d690c5d516abab` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:18.310532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca11529-0da8-459f-9c4a-9530a68e91e0 None admin] RBAC: Authorizing `identity:get_user(user_id=5866c6e8a00d4826b8d690c5d516abab)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:18.312037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca11529-0da8-459f-9c4a-9530a68e91e0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:18.314072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 154/308] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:18 2022] GET /identity/v3/users/5866c6e8a00d4826b8d690c5d516abab => generated 293 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:18.319888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca11529-0da8-459f-9c4a-9530a68e91e0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:18.328900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 155/309] 10.222.0.25 () {58 vars in 1334 bytes} [Fri Jun 10 12:45:18 2022] PUT /identity/v3/system/users/1f755d0a272845638e015f44d42de665/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:18.395106 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:18.433775 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca11529-0da8-459f-9c4a-9530a68e91e0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:18.474043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dbe8e39-af4e-4060-a408-70d6b063a9f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:18.474714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbe8e39-af4e-4060-a408-70d6b063a9f7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:18.475001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbe8e39-af4e-4060-a408-70d6b063a9f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:18.475281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dbe8e39-af4e-4060-a408-70d6b063a9f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:18.476132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbe8e39-af4e-4060-a408-70d6b063a9f7 None admin] RBAC: Adding query filter params (role.id=94339d681b5c4fd188991c590c897ad4, user.id=5866c6e8a00d4826b8d690c5d516abab, scope.system=all) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:18.476500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbe8e39-af4e-4060-a408-70d6b063a9f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:18.477939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dbe8e39-af4e-4060-a408-70d6b063a9f7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:18.484406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 155/310] 10.222.0.25 () {56 vars in 1308 bytes} [Fri Jun 10 12:45:18 2022] GET /identity/v3/role_assignments?user.id=5866c6e8a00d4826b8d690c5d516abab&scope.system=all&role.id=94339d681b5c4fd188991c590c897ad4 => generated 226 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:18.696162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd99755-a43e-44f5-8679-d598988d50ed None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:18.696524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd99755-a43e-44f5-8679-d598988d50ed None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:18.696735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd99755-a43e-44f5-8679-d598988d50ed None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:18.698827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 156/311] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:18.704279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec41d36-8bd6-4438-ad20-ec5bff13e3ea None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:18.704823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec41d36-8bd6-4438-ad20-ec5bff13e3ea None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:18.705297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cec41d36-8bd6-4438-ad20-ec5bff13e3ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:18.722315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-cec41d36-8bd6-4438-ad20-ec5bff13e3ea None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:18.723855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-cec41d36-8bd6-4438-ad20-ec5bff13e3ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:18.797121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 156/312] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:18.820410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd99755-a43e-44f5-8679-d598988d50ed None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:18.931010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd99755-a43e-44f5-8679-d598988d50ed None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:18.970677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fa6b18-2dd4-4ee3-8c66-1d258a1b5b68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:18.971478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fa6b18-2dd4-4ee3-8c66-1d258a1b5b68 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:18.971770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fa6b18-2dd4-4ee3-8c66-1d258a1b5b68 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:18.972048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3fa6b18-2dd4-4ee3-8c66-1d258a1b5b68 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:18.982736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3fa6b18-2dd4-4ee3-8c66-1d258a1b5b68 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-c3fa6b18-2dd4-4ee3-8c66-1d258a1b5b68 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:18.990502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:18.991372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:18.992533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 157/313] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:18 2022] GET /identity/v3/roles/reader => generated 84 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:18.999589 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec41d36-8bd6-4438-ad20-ec5bff13e3ea None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:19.062814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:19.101808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cec41d36-8bd6-4438-ad20-ec5bff13e3ea None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:19.135715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb39926-00e5-4284-a61b-5112bcea80d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:19.136403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb39926-00e5-4284-a61b-5112bcea80d2 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:19.137020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb39926-00e5-4284-a61b-5112bcea80d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:19.137302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb39926-00e5-4284-a61b-5112bcea80d2 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:19.138069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb39926-00e5-4284-a61b-5112bcea80d2 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:19.138377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb39926-00e5-4284-a61b-5112bcea80d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:19.139800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb39926-00e5-4284-a61b-5112bcea80d2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:19.149114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 157/314] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:18 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:19.155334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fa6b18-2dd4-4ee3-8c66-1d258a1b5b68 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:19.197002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:19.270393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3fa6b18-2dd4-4ee3-8c66-1d258a1b5b68 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:19.304595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb0b13a-94e6-428a-aa1c-24fb84c66156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:19.305555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb0b13a-94e6-428a-aa1c-24fb84c66156 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:19.305844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb0b13a-94e6-428a-aa1c-24fb84c66156 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:19.306122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb0b13a-94e6-428a-aa1c-24fb84c66156 None admin] PATH_INFO: `/v3/users/f471a18fc01a4fb2a2df6a3f86749926` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:19.307872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb0b13a-94e6-428a-aa1c-24fb84c66156 None admin] RBAC: Authorizing `identity:get_user(user_id=f471a18fc01a4fb2a2df6a3f86749926)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:19.309359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb0b13a-94e6-428a-aa1c-24fb84c66156 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:19.311175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 158/315] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:19 2022] GET /identity/v3/users/f471a18fc01a4fb2a2df6a3f86749926 => generated 297 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:19.316542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb39926-00e5-4284-a61b-5112bcea80d2 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:19.421844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb39926-00e5-4284-a61b-5112bcea80d2 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:19.445185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:19.456648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2a402-ca1b-4dcd-8e44-c010be029732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:19.457365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2a402-ca1b-4dcd-8e44-c010be029732 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:19.457650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2a402-ca1b-4dcd-8e44-c010be029732 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:19.457929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb2a402-ca1b-4dcd-8e44-c010be029732 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:19.528284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb2a402-ca1b-4dcd-8e44-c010be029732 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:19.530066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddb2a402-ca1b-4dcd-8e44-c010be029732 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:19.629412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 158/316] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jun 10 12:45:19.635543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb0b13a-94e6-428a-aa1c-24fb84c66156 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:19.740569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb0b13a-94e6-428a-aa1c-24fb84c66156 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:19.743059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:19.774346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6dffd3-ee1c-4542-9d64-a0e7226af591 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:19.775208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6dffd3-ee1c-4542-9d64-a0e7226af591 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:19.775496 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6dffd3-ee1c-4542-9d64-a0e7226af591 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:19.776290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6dffd3-ee1c-4542-9d64-a0e7226af591 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:19.777507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6dffd3-ee1c-4542-9d64-a0e7226af591 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:19.779232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6dffd3-ee1c-4542-9d64-a0e7226af591 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:19.781263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 159/317] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:19 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:19.786979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2a402-ca1b-4dcd-8e44-c010be029732 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:19.813157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:19.887891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb2a402-ca1b-4dcd-8e44-c010be029732 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:19.923000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae9718e-b222-4ff8-9e49-17b7fe5b3119 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:19.923655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9718e-b222-4ff8-9e49-17b7fe5b3119 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:19.923941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9718e-b222-4ff8-9e49-17b7fe5b3119 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:19.924220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae9718e-b222-4ff8-9e49-17b7fe5b3119 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:19.925078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9718e-b222-4ff8-9e49-17b7fe5b3119 None admin] RBAC: Adding query filter params (role.id=94339d681b5c4fd188991c590c897ad4, scope.project.id=b77e955a802a489daa9bd13993979e40, user.id=f471a18fc01a4fb2a2df6a3f86749926) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:19.925389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9718e-b222-4ff8-9e49-17b7fe5b3119 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:19.926829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae9718e-b222-4ff8-9e49-17b7fe5b3119 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:19.933020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 159/318] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:19 2022] GET /identity/v3/role_assignments?user.id=f471a18fc01a4fb2a2df6a3f86749926&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=94339d681b5c4fd188991c590c897ad4 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:20.481685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dfa1e0-0ada-4f08-8839-187756096c59 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:20.482232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dfa1e0-0ada-4f08-8839-187756096c59 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:20.482232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50dfa1e0-0ada-4f08-8839-187756096c59 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:20.483896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 160/319] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:20.489410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423c1d60-9667-471b-b26a-1909601de589 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:20.489892 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423c1d60-9667-471b-b26a-1909601de589 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:20.489993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423c1d60-9667-471b-b26a-1909601de589 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:20.507442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-423c1d60-9667-471b-b26a-1909601de589 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:20.508626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-423c1d60-9667-471b-b26a-1909601de589 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:20.538478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be90fed-bae7-47d0-8b51-66f8325dcb22 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:20.538778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be90fed-bae7-47d0-8b51-66f8325dcb22 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:20.539028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be90fed-bae7-47d0-8b51-66f8325dcb22 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:20.540536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 161/320] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:45:20.545799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180dec26-00e4-4d32-b019-92b9d74010ab None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:20.546097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180dec26-00e4-4d32-b019-92b9d74010ab None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:20.546380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180dec26-00e4-4d32-b019-92b9d74010ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:20.563164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-180dec26-00e4-4d32-b019-92b9d74010ab None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:20.564345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-180dec26-00e4-4d32-b019-92b9d74010ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:20.579926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 160/321] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:20.602535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423c1d60-9667-471b-b26a-1909601de589 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:20.623239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:20.645153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 162/322] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:20.675953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180dec26-00e4-4d32-b019-92b9d74010ab None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:20.718371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-423c1d60-9667-471b-b26a-1909601de589 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:20.752579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971405ea-3f92-4ee1-bdee-cb4842e4674f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:20.753389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971405ea-3f92-4ee1-bdee-cb4842e4674f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:20.753676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971405ea-3f92-4ee1-bdee-cb4842e4674f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:20.753956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971405ea-3f92-4ee1-bdee-cb4842e4674f None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:20.755118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971405ea-3f92-4ee1-bdee-cb4842e4674f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:20.756575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971405ea-3f92-4ee1-bdee-cb4842e4674f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:20.758224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 161/323] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:20.759381 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7473751f-a417-48b7-bf9d-bf48e9b905c7 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:20.759659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7473751f-a417-48b7-bf9d-bf48e9b905c7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:20.759970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7473751f-a417-48b7-bf9d-bf48e9b905c7 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:20.764953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 162/324] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:20 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:20.764953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7473751f-a417-48b7-bf9d-bf48e9b905c7 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:20.784142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180dec26-00e4-4d32-b019-92b9d74010ab None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:20.786105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:20.818577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3d86bd-1429-4e0a-a935-f5b3549ef64e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:20.819371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3d86bd-1429-4e0a-a935-f5b3549ef64e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:20.819740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3d86bd-1429-4e0a-a935-f5b3549ef64e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:20.819936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3d86bd-1429-4e0a-a935-f5b3549ef64e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:20.830080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3d86bd-1429-4e0a-a935-f5b3549ef64e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-8b3d86bd-1429-4e0a-a935-f5b3549ef64e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:20.837734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:20.839139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:20.839927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 163/325] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:20.841417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8743fd00-eb4f-4a55-bc48-cbc8409eb3a4 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:20.841417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8743fd00-eb4f-4a55-bc48-cbc8409eb3a4 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:20.841417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8743fd00-eb4f-4a55-bc48-cbc8409eb3a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:20.849004 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:20.858318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-8743fd00-eb4f-4a55-bc48-cbc8409eb3a4 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:20.859433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8743fd00-eb4f-4a55-bc48-cbc8409eb3a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:20.864417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7473751f-a417-48b7-bf9d-bf48e9b905c7 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:20.898427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95598550-bfd9-4747-8629-44dc1c01226b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:20.899065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95598550-bfd9-4747-8629-44dc1c01226b None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:20.899348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95598550-bfd9-4747-8629-44dc1c01226b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:20.899626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95598550-bfd9-4747-8629-44dc1c01226b None admin] PATH_INFO: `/v3/groups` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:20.900307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95598550-bfd9-4747-8629-44dc1c01226b None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:20.902335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95598550-bfd9-4747-8629-44dc1c01226b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:20.926675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 164/326] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:20.927509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8743fd00-eb4f-4a55-bc48-cbc8409eb3a4 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:20.983500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 163/327] 10.222.0.25 () {60 vars in 1134 bytes} [Fri Jun 10 12:45:20 2022] POST /identity/v3/groups => generated 226 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:45:20.984601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95598550-bfd9-4747-8629-44dc1c01226b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:21.045079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:21.055465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8743fd00-eb4f-4a55-bc48-cbc8409eb3a4 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:21.094975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5416287b-b7b0-4f34-b80b-f6c7c92952d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:21.095933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5416287b-b7b0-4f34-b80b-f6c7c92952d5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:21.096219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5416287b-b7b0-4f34-b80b-f6c7c92952d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:21.096496 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5416287b-b7b0-4f34-b80b-f6c7c92952d5 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:21.097476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5416287b-b7b0-4f34-b80b-f6c7c92952d5 None admin] RBAC: Adding query filter params (name=member) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:21.098622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5416287b-b7b0-4f34-b80b-f6c7c92952d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:21.100105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5416287b-b7b0-4f34-b80b-f6c7c92952d5 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:21.116460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 165/328] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:21.118215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5416287b-b7b0-4f34-b80b-f6c7c92952d5 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:21.129650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95598550-bfd9-4747-8629-44dc1c01226b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:21.173551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009c4144-f8a4-4d39-9b71-336802265613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:21.174353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009c4144-f8a4-4d39-9b71-336802265613 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:21.174641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009c4144-f8a4-4d39-9b71-336802265613 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:21.174919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009c4144-f8a4-4d39-9b71-336802265613 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:21.186993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009c4144-f8a4-4d39-9b71-336802265613 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-009c4144-f8a4-4d39-9b71-336802265613 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:21.194940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:21.195823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:21.196738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 164/329] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:20 2022] GET /identity/v3/roles/reader => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:21.202313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009c4144-f8a4-4d39-9b71-336802265613 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:21.211459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:21.232729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5416287b-b7b0-4f34-b80b-f6c7c92952d5 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:21.282322 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0c845a-80f6-4a46-975e-fb7c7133fe20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:21.283193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0c845a-80f6-4a46-975e-fb7c7133fe20 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:21.283475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0c845a-80f6-4a46-975e-fb7c7133fe20 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:21.283752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe0c845a-80f6-4a46-975e-fb7c7133fe20 None admin] PATH_INFO: `/v3/users/1f755d0a272845638e015f44d42de665` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:21.285090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0c845a-80f6-4a46-975e-fb7c7133fe20 None admin] RBAC: Authorizing `identity:get_user(user_id=1f755d0a272845638e015f44d42de665)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:21.286570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe0c845a-80f6-4a46-975e-fb7c7133fe20 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:21.292674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 166/330] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:21 2022] GET /identity/v3/users/1f755d0a272845638e015f44d42de665 => generated 293 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:21.294878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0c845a-80f6-4a46-975e-fb7c7133fe20 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:21.325700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:21.331929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009c4144-f8a4-4d39-9b71-336802265613 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:21.379329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48427639-af80-41c2-b727-ec4d5ff63566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:21.379936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48427639-af80-41c2-b727-ec4d5ff63566 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:21.380229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48427639-af80-41c2-b727-ec4d5ff63566 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:21.380497 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48427639-af80-41c2-b727-ec4d5ff63566 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:21.381390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48427639-af80-41c2-b727-ec4d5ff63566 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:21.382321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48427639-af80-41c2-b727-ec4d5ff63566 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:21.383020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48427639-af80-41c2-b727-ec4d5ff63566 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:21.393906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 165/331] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:21.399236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48427639-af80-41c2-b727-ec4d5ff63566 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:21.415261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe0c845a-80f6-4a46-975e-fb7c7133fe20 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:21.460040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76233f39-bef4-44be-be57-cb9991b05690 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:21.460702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76233f39-bef4-44be-be57-cb9991b05690 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:21.461027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76233f39-bef4-44be-be57-cb9991b05690 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:21.461303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76233f39-bef4-44be-be57-cb9991b05690 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:21.462457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76233f39-bef4-44be-be57-cb9991b05690 None admin] RBAC: Adding query filter params (role.id=dfc4731db42b4f0bbb5e10d84814f9cd, user.id=1f755d0a272845638e015f44d42de665, scope.system=all) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:21.462770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76233f39-bef4-44be-be57-cb9991b05690 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:21.464812 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76233f39-bef4-44be-be57-cb9991b05690 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:21.481848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 167/332] 10.222.0.25 () {56 vars in 1308 bytes} [Fri Jun 10 12:45:21 2022] GET /identity/v3/role_assignments?user.id=1f755d0a272845638e015f44d42de665&scope.system=all&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 510 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:21.512951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:21.538790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48427639-af80-41c2-b727-ec4d5ff63566 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:21.579193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff09bea6-8a22-4b4e-bd51-dabf13c997c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:21.580214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff09bea6-8a22-4b4e-bd51-dabf13c997c4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:21.580501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff09bea6-8a22-4b4e-bd51-dabf13c997c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:21.580808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff09bea6-8a22-4b4e-bd51-dabf13c997c4 None admin] PATH_INFO: `/v3/users/5866c6e8a00d4826b8d690c5d516abab` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:21.582163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff09bea6-8a22-4b4e-bd51-dabf13c997c4 None admin] RBAC: Authorizing `identity:get_user(user_id=5866c6e8a00d4826b8d690c5d516abab)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:21.583705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff09bea6-8a22-4b4e-bd51-dabf13c997c4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:21.585845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 166/333] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:21 2022] GET /identity/v3/users/5866c6e8a00d4826b8d690c5d516abab => generated 293 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:21.591624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76233f39-bef4-44be-be57-cb9991b05690 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:21.687678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:21.708148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76233f39-bef4-44be-be57-cb9991b05690 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:21.744530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888cf89a-0f0f-449c-a46a-2d6558781a46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:21.745475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888cf89a-0f0f-449c-a46a-2d6558781a46 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:21.745739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888cf89a-0f0f-449c-a46a-2d6558781a46 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:21.746431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888cf89a-0f0f-449c-a46a-2d6558781a46 None admin] PATH_INFO: `/v3/system/users/5866c6e8a00d4826b8d690c5d516abab/roles/94339d681b5c4fd188991c590c897ad4` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:21.748082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888cf89a-0f0f-449c-a46a-2d6558781a46 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=5866c6e8a00d4826b8d690c5d516abab, role_id=94339d681b5c4fd188991c590c897ad4)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:21.750137 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-888cf89a-0f0f-449c-a46a-2d6558781a46 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:21.832286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 168/334] 10.222.0.25 () {58 vars in 1334 bytes} [Fri Jun 10 12:45:21 2022] PUT /identity/v3/system/users/5866c6e8a00d4826b8d690c5d516abab/roles/94339d681b5c4fd188991c590c897ad4 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:23.663245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610797a-3a65-406f-87df-c468500a870e None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:23.663727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610797a-3a65-406f-87df-c468500a870e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:23.663727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6610797a-3a65-406f-87df-c468500a870e None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:23.665462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 167/335] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:23.671200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba8cfe9-cbf4-4f95-ac50-1636946bafc3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:23.671632 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba8cfe9-cbf4-4f95-ac50-1636946bafc3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:23.671825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba8cfe9-cbf4-4f95-ac50-1636946bafc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:23.697497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5ba8cfe9-cbf4-4f95-ac50-1636946bafc3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:23.698741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5ba8cfe9-cbf4-4f95-ac50-1636946bafc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:23.781419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 169/336] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:23.824041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610797a-3a65-406f-87df-c468500a870e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:23.968336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6610797a-3a65-406f-87df-c468500a870e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:24.009621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23c00a0-3d3f-4e6d-a486-1c2a6279a383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:24.011307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23c00a0-3d3f-4e6d-a486-1c2a6279a383 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.011837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23c00a0-3d3f-4e6d-a486-1c2a6279a383 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.012308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23c00a0-3d3f-4e6d-a486-1c2a6279a383 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.020958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:24.029755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23c00a0-3d3f-4e6d-a486-1c2a6279a383 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-f23c00a0-3d3f-4e6d-a486-1c2a6279a383 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:24.038972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:24.039904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:24.040676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:24.048046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 168/337] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:23 2022] GET /identity/v3/roles/member => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:24.051202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba8cfe9-cbf4-4f95-ac50-1636946bafc3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:24.053143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29574a1-0cd5-4292-adfd-bdfa3ac047eb None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.053637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29574a1-0cd5-4292-adfd-bdfa3ac047eb None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.054059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29574a1-0cd5-4292-adfd-bdfa3ac047eb None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.059301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 169/338] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:24.061341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a516e1-3410-4a6d-a233-53aa6e20ee1b None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.061780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a516e1-3410-4a6d-a233-53aa6e20ee1b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.062223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a516e1-3410-4a6d-a233-53aa6e20ee1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.083573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-01a516e1-3410-4a6d-a233-53aa6e20ee1b None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:24.085038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-01a516e1-3410-4a6d-a233-53aa6e20ee1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:24.188481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:24.200874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 170/339] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:24.222966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba8cfe9-cbf4-4f95-ac50-1636946bafc3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:24.266630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939a998a-a2aa-4bbb-9639-883952e2bc6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:24.267316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939a998a-a2aa-4bbb-9639-883952e2bc6c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.267606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939a998a-a2aa-4bbb-9639-883952e2bc6c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.267889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939a998a-a2aa-4bbb-9639-883952e2bc6c None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.268647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939a998a-a2aa-4bbb-9639-883952e2bc6c None admin] RBAC: Adding query filter params (name=member) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:24.268988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939a998a-a2aa-4bbb-9639-883952e2bc6c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:24.272052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939a998a-a2aa-4bbb-9639-883952e2bc6c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:24.274625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a516e1-3410-4a6d-a233-53aa6e20ee1b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:24.295633 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 170/340] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:24.298729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939a998a-a2aa-4bbb-9639-883952e2bc6c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:24.461048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a516e1-3410-4a6d-a233-53aa6e20ee1b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:24.478285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939a998a-a2aa-4bbb-9639-883952e2bc6c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:24.521665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:24.527507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a19cea9-1a4e-4d50-b332-3ed2f4d985fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:24.529180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a19cea9-1a4e-4d50-b332-3ed2f4d985fa None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.529180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a19cea9-1a4e-4d50-b332-3ed2f4d985fa None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.529180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a19cea9-1a4e-4d50-b332-3ed2f4d985fa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.536111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:24.546090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98774d3-6e74-4915-be07-4c87b5f755bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:24.546932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98774d3-6e74-4915-be07-4c87b5f755bd None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.547217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98774d3-6e74-4915-be07-4c87b5f755bd None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.547495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98774d3-6e74-4915-be07-4c87b5f755bd None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.578385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98774d3-6e74-4915-be07-4c87b5f755bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-b98774d3-6e74-4915-be07-4c87b5f755bd None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:24.586551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:24.587447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:24.588827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 171/341] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:24.603550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52add002-e9bb-4817-9918-f2f30cd2beb7 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.603854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52add002-e9bb-4817-9918-f2f30cd2beb7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.604113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52add002-e9bb-4817-9918-f2f30cd2beb7 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.605737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 172/342] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:24 2022] GET /identity => generated 270 bytes in 17 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:24.606279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52add002-e9bb-4817-9918-f2f30cd2beb7 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:24.652494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a19cea9-1a4e-4d50-b332-3ed2f4d985fa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:24.654317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a19cea9-1a4e-4d50-b332-3ed2f4d985fa None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:24.779450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52add002-e9bb-4817-9918-f2f30cd2beb7 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:24.787257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 171/343] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 489 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:24.789095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c931efc1-c07f-40ee-98e8-a1f3b31b0634 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.789615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c931efc1-c07f-40ee-98e8-a1f3b31b0634 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.790105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c931efc1-c07f-40ee-98e8-a1f3b31b0634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.807165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-c931efc1-c07f-40ee-98e8-a1f3b31b0634 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:24.808666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c931efc1-c07f-40ee-98e8-a1f3b31b0634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:24.821782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e685f9-3424-4236-b81a-c27a95ae7da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:24.822998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e685f9-3424-4236-b81a-c27a95ae7da2 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.823276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e685f9-3424-4236-b81a-c27a95ae7da2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.823549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e685f9-3424-4236-b81a-c27a95ae7da2 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.824400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e685f9-3424-4236-b81a-c27a95ae7da2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:24.824707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e685f9-3424-4236-b81a-c27a95ae7da2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:24.826225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e685f9-3424-4236-b81a-c27a95ae7da2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:24.838227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 173/344] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:24 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:24.839687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7a677d-efd8-491f-a500-ff387a9e2b48 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:24.839977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7a677d-efd8-491f-a500-ff387a9e2b48 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:24.840253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7a677d-efd8-491f-a500-ff387a9e2b48 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:24.841641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 174/345] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:24.842182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7a677d-efd8-491f-a500-ff387a9e2b48 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:24.846696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:24.888459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 172/346] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:24.889695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c931efc1-c07f-40ee-98e8-a1f3b31b0634 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:24.919385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:25.011602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7a677d-efd8-491f-a500-ff387a9e2b48 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.019158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c931efc1-c07f-40ee-98e8-a1f3b31b0634 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.052737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df0efaa-9f21-42fd-a740-4471d8bd4a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.053713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df0efaa-9f21-42fd-a740-4471d8bd4a62 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.053918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df0efaa-9f21-42fd-a740-4471d8bd4a62 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.054193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df0efaa-9f21-42fd-a740-4471d8bd4a62 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.055596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df0efaa-9f21-42fd-a740-4471d8bd4a62 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:25.057728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3df0efaa-9f21-42fd-a740-4471d8bd4a62 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:25.058493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5f048d-de3f-4ca5-98f6-bedc83f79f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.059414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f048d-de3f-4ca5-98f6-bedc83f79f1d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.059732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 175/347] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:24 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:25.060141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f048d-de3f-4ca5-98f6-bedc83f79f1d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.060616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f048d-de3f-4ca5-98f6-bedc83f79f1d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.060950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccf4e75-8687-4645-99b1-00ff2e6f050f None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.061232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccf4e75-8687-4645-99b1-00ff2e6f050f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.061505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccf4e75-8687-4645-99b1-00ff2e6f050f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.077950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-9ccf4e75-8687-4645-99b1-00ff2e6f050f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:25.079443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9ccf4e75-8687-4645-99b1-00ff2e6f050f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:25.099997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:25.136497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5f048d-de3f-4ca5-98f6-bedc83f79f1d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:25.138356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb5f048d-de3f-4ca5-98f6-bedc83f79f1d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:25.155496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 176/348] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:25.156357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccf4e75-8687-4645-99b1-00ff2e6f050f None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:25.206963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:25.266473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccf4e75-8687-4645-99b1-00ff2e6f050f None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.279651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 173/349] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:25.280665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5f048d-de3f-4ca5-98f6-bedc83f79f1d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:25.304902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87d56e6-7062-48a0-a68c-cce160297903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.305908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87d56e6-7062-48a0-a68c-cce160297903 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.306308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87d56e6-7062-48a0-a68c-cce160297903 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.306732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c87d56e6-7062-48a0-a68c-cce160297903 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.317519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c87d56e6-7062-48a0-a68c-cce160297903 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c87d56e6-7062-48a0-a68c-cce160297903 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:25.326292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:25.327308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:25.328105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:25.330826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 177/350] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:25.331814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87d56e6-7062-48a0-a68c-cce160297903 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:25.373158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:25.405448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5f048d-de3f-4ca5-98f6-bedc83f79f1d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.441871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c87d56e6-7062-48a0-a68c-cce160297903 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.442853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f30aec-08e8-4bf5-a991-612c8b07a70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.443904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f30aec-08e8-4bf5-a991-612c8b07a70c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.444388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f30aec-08e8-4bf5-a991-612c8b07a70c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.444752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f30aec-08e8-4bf5-a991-612c8b07a70c None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.450846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:25.453526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f30aec-08e8-4bf5-a991-612c8b07a70c None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:25.455249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f30aec-08e8-4bf5-a991-612c8b07a70c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:25.457184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 174/351] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:25 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:25.458280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f30aec-08e8-4bf5-a991-612c8b07a70c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:25.480326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cee6e0-e445-4ab9-9917-c472707d0e35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.481264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cee6e0-e445-4ab9-9917-c472707d0e35 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.481893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cee6e0-e445-4ab9-9917-c472707d0e35 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.482483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cee6e0-e445-4ab9-9917-c472707d0e35 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.494532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cee6e0-e445-4ab9-9917-c472707d0e35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c0cee6e0-e445-4ab9-9917-c472707d0e35 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:25.504848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:25.505825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:25.506622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 178/352] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:25.508005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cee6e0-e445-4ab9-9917-c472707d0e35 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:25.581983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f30aec-08e8-4bf5-a991-612c8b07a70c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.613556 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:25.617778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cee6e0-e445-4ab9-9917-c472707d0e35 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.622017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83ed0b3-d5ee-47d7-80eb-3b2e7ab4420c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.623223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83ed0b3-d5ee-47d7-80eb-3b2e7ab4420c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.623703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83ed0b3-d5ee-47d7-80eb-3b2e7ab4420c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.624171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83ed0b3-d5ee-47d7-80eb-3b2e7ab4420c None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.625888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83ed0b3-d5ee-47d7-80eb-3b2e7ab4420c None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:25.627845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83ed0b3-d5ee-47d7-80eb-3b2e7ab4420c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:25.630555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 175/353] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:25 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:25.631726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83ed0b3-d5ee-47d7-80eb-3b2e7ab4420c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:25.661573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6fae43-5efb-474e-a076-50f1c8298c80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.662309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6fae43-5efb-474e-a076-50f1c8298c80 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.662592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6fae43-5efb-474e-a076-50f1c8298c80 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.662867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6fae43-5efb-474e-a076-50f1c8298c80 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.663654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6fae43-5efb-474e-a076-50f1c8298c80 None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:25.664005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6fae43-5efb-474e-a076-50f1c8298c80 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:25.665538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6fae43-5efb-474e-a076-50f1c8298c80 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:25.675631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 179/354] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:25.676524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6fae43-5efb-474e-a076-50f1c8298c80 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:25.710805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:25.756390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83ed0b3-d5ee-47d7-80eb-3b2e7ab4420c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.793676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc6b787-b27c-4206-923e-2dd27f116f25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.794329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc6b787-b27c-4206-923e-2dd27f116f25 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.794612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc6b787-b27c-4206-923e-2dd27f116f25 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.794886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc6b787-b27c-4206-923e-2dd27f116f25 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.795365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6fae43-5efb-474e-a076-50f1c8298c80 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.795709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc6b787-b27c-4206-923e-2dd27f116f25 None admin] RBAC: Adding query filter params (group.id=9c84270a862240469756e6d3d3dd760a, role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:25.796019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc6b787-b27c-4206-923e-2dd27f116f25 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:25.797508 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc6b787-b27c-4206-923e-2dd27f116f25 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:25.803205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 176/355] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:25 2022] GET /identity/v3/role_assignments?group.id=9c84270a862240469756e6d3d3dd760a&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 260 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:25.804180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc6b787-b27c-4206-923e-2dd27f116f25 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:25.842469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7af6029-1135-40cf-b053-0e61e8b67913 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.842974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7af6029-1135-40cf-b053-0e61e8b67913 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.843260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7af6029-1135-40cf-b053-0e61e8b67913 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.843541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7af6029-1135-40cf-b053-0e61e8b67913 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.844301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7af6029-1135-40cf-b053-0e61e8b67913 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:25.844612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7af6029-1135-40cf-b053-0e61e8b67913 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:25.846085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7af6029-1135-40cf-b053-0e61e8b67913 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:25.859160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 180/356] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:25.859160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7af6029-1135-40cf-b053-0e61e8b67913 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:25.885082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:25.934282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc6b787-b27c-4206-923e-2dd27f116f25 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:25.990055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:25.990384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf254469-5218-4cab-89aa-94ddb3c633a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:25.992916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf254469-5218-4cab-89aa-94ddb3c633a5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:25.992916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf254469-5218-4cab-89aa-94ddb3c633a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:25.992916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf254469-5218-4cab-89aa-94ddb3c633a5 None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:25.994638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf254469-5218-4cab-89aa-94ddb3c633a5 None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:25.996729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf254469-5218-4cab-89aa-94ddb3c633a5 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:25.998759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 177/357] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:25 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:26.002082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee781b64-4c63-404c-9685-df262a755c2d None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.002369 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee781b64-4c63-404c-9685-df262a755c2d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.002648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee781b64-4c63-404c-9685-df262a755c2d None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.004126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 178/358] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:26.005054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cd62d4-adb5-40c7-bdff-2cd85c29a930 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.005337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cd62d4-adb5-40c7-bdff-2cd85c29a930 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.005613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35cd62d4-adb5-40c7-bdff-2cd85c29a930 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.007054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 179/359] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:26.014202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cd62d4-adb5-40c7-bdff-2cd85c29a930 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:26.014328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7af6029-1135-40cf-b053-0e61e8b67913 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:26.055861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733fe91e-ed87-4a42-b352-0d83986c6df2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:26.056566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733fe91e-ed87-4a42-b352-0d83986c6df2 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.056959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733fe91e-ed87-4a42-b352-0d83986c6df2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.057244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733fe91e-ed87-4a42-b352-0d83986c6df2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.137917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733fe91e-ed87-4a42-b352-0d83986c6df2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:26.139299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733fe91e-ed87-4a42-b352-0d83986c6df2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:26.139425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35cd62d4-adb5-40c7-bdff-2cd85c29a930 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:26.177729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:26.185657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483cee45-92bb-45a4-964f-6fe293171596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:26.186536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cee45-92bb-45a4-964f-6fe293171596 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.187533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cee45-92bb-45a4-964f-6fe293171596 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.187533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-483cee45-92bb-45a4-964f-6fe293171596 None admin] PATH_INFO: `/v3/users/5866c6e8a00d4826b8d690c5d516abab` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.189294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483cee45-92bb-45a4-964f-6fe293171596 None admin] RBAC: Authorizing `identity:get_user(user_id=5866c6e8a00d4826b8d690c5d516abab)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:26.190929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-483cee45-92bb-45a4-964f-6fe293171596 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:26.192667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 180/360] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:45:26 2022] GET /identity/v3/users/5866c6e8a00d4826b8d690c5d516abab => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:26.193731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483cee45-92bb-45a4-964f-6fe293171596 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:26.258948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:26.276410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 181/361] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:26.277764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bbbe64-6360-4cf3-852a-3242f69cf98c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.278018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bbbe64-6360-4cf3-852a-3242f69cf98c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.278554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bbbe64-6360-4cf3-852a-3242f69cf98c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.298972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-70bbbe64-6360-4cf3-852a-3242f69cf98c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:26.300166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-70bbbe64-6360-4cf3-852a-3242f69cf98c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:26.329706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-483cee45-92bb-45a4-964f-6fe293171596 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:26.374303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-137ad512-a61c-451c-a01c-81e7e1d018b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:26.375527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137ad512-a61c-451c-a01c-81e7e1d018b4 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.375811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137ad512-a61c-451c-a01c-81e7e1d018b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.376093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-137ad512-a61c-451c-a01c-81e7e1d018b4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.377259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137ad512-a61c-451c-a01c-81e7e1d018b4 None admin] RBAC: Adding query filter params (group.id=9c84270a862240469756e6d3d3dd760a, role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:26.377581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137ad512-a61c-451c-a01c-81e7e1d018b4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:26.379273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-137ad512-a61c-451c-a01c-81e7e1d018b4 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:26.386822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 181/362] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:26 2022] GET /identity/v3/role_assignments?group.id=9c84270a862240469756e6d3d3dd760a&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 260 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:26.388270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575c5f03-f782-4a8f-b633-204b1c7ff758 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.388510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575c5f03-f782-4a8f-b633-204b1c7ff758 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.388799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575c5f03-f782-4a8f-b633-204b1c7ff758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.399966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 182/363] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:26.401212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bbbe64-6360-4cf3-852a-3242f69cf98c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:26.411254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-575c5f03-f782-4a8f-b633-204b1c7ff758 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:26.412507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-575c5f03-f782-4a8f-b633-204b1c7ff758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:26.499134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:26.511768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 182/364] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:26.512698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575c5f03-f782-4a8f-b633-204b1c7ff758 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:26.552306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bbbe64-6360-4cf3-852a-3242f69cf98c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:26.579121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:26.600278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab8b41-e379-4eae-8142-0273dbf9dbb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:26.600981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab8b41-e379-4eae-8142-0273dbf9dbb4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.601264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab8b41-e379-4eae-8142-0273dbf9dbb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.601731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cab8b41-e379-4eae-8142-0273dbf9dbb4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.603062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cab8b41-e379-4eae-8142-0273dbf9dbb4 None admin] RBAC: Adding query filter params (role.id=94339d681b5c4fd188991c590c897ad4, user.id=5866c6e8a00d4826b8d690c5d516abab, scope.system=all) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:26.603431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cab8b41-e379-4eae-8142-0273dbf9dbb4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:26.604911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cab8b41-e379-4eae-8142-0273dbf9dbb4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:26.618583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 183/365] 10.222.0.25 () {56 vars in 1308 bytes} [Fri Jun 10 12:45:26 2022] GET /identity/v3/role_assignments?user.id=5866c6e8a00d4826b8d690c5d516abab&scope.system=all&role.id=94339d681b5c4fd188991c590c897ad4 => generated 510 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:26.620088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab8b41-e379-4eae-8142-0273dbf9dbb4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:26.672065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575c5f03-f782-4a8f-b633-204b1c7ff758 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:26.714279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067541c3-f13e-45db-a6d2-3e31181ca694 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:26.715195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067541c3-f13e-45db-a6d2-3e31181ca694 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.715482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067541c3-f13e-45db-a6d2-3e31181ca694 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.715763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067541c3-f13e-45db-a6d2-3e31181ca694 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.717081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067541c3-f13e-45db-a6d2-3e31181ca694 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:26.718574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067541c3-f13e-45db-a6d2-3e31181ca694 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:26.720696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 183/366] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:26 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:26.721552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067541c3-f13e-45db-a6d2-3e31181ca694 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:26.757264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cab8b41-e379-4eae-8142-0273dbf9dbb4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:26.790666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:26.796812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618fa1d0-d534-486f-86e7-c5fec352d5d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:26.797605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618fa1d0-d534-486f-86e7-c5fec352d5d4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.804429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618fa1d0-d534-486f-86e7-c5fec352d5d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.804429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618fa1d0-d534-486f-86e7-c5fec352d5d4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.810749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618fa1d0-d534-486f-86e7-c5fec352d5d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-618fa1d0-d534-486f-86e7-c5fec352d5d4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:26.819122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:26.820949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 184/367] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:45:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:26.822875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618fa1d0-d534-486f-86e7-c5fec352d5d4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:26.839170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067541c3-f13e-45db-a6d2-3e31181ca694 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:26.866456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:26.878675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10484017-c351-4762-865e-b08bfb935363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:26.879639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10484017-c351-4762-865e-b08bfb935363 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.879926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10484017-c351-4762-865e-b08bfb935363 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.880206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10484017-c351-4762-865e-b08bfb935363 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.893407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10484017-c351-4762-865e-b08bfb935363 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-10484017-c351-4762-865e-b08bfb935363 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:26.901771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:26.904455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 184/368] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:26.906375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10484017-c351-4762-865e-b08bfb935363 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:26.935950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618fa1d0-d534-486f-86e7-c5fec352d5d4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:26.972983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99da43ea-3811-4815-a54e-0f21fa2c8bea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:26.973852 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99da43ea-3811-4815-a54e-0f21fa2c8bea None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:26.974132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99da43ea-3811-4815-a54e-0f21fa2c8bea None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:26.974402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99da43ea-3811-4815-a54e-0f21fa2c8bea None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:26.975635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99da43ea-3811-4815-a54e-0f21fa2c8bea None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:26.977165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99da43ea-3811-4815-a54e-0f21fa2c8bea None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:26.979399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 185/369] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:26 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:26.981174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99da43ea-3811-4815-a54e-0f21fa2c8bea None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:27.017881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10484017-c351-4762-865e-b08bfb935363 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:27.038923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:27.056087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40f0cc3-9ed1-4c09-8cea-130ee509aa74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:27.057375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40f0cc3-9ed1-4c09-8cea-130ee509aa74 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:27.057663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40f0cc3-9ed1-4c09-8cea-130ee509aa74 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:27.057941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40f0cc3-9ed1-4c09-8cea-130ee509aa74 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:27.058704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40f0cc3-9ed1-4c09-8cea-130ee509aa74 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:27.059016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40f0cc3-9ed1-4c09-8cea-130ee509aa74 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:27.060419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40f0cc3-9ed1-4c09-8cea-130ee509aa74 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:27.070278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 185/370] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:27.070943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40f0cc3-9ed1-4c09-8cea-130ee509aa74 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:27.093904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99da43ea-3811-4815-a54e-0f21fa2c8bea None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:27.101432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:27.130506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a260083-a1c8-4d6b-b8fa-4a7d03be49f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:27.131411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a260083-a1c8-4d6b-b8fa-4a7d03be49f5 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:27.131883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a260083-a1c8-4d6b-b8fa-4a7d03be49f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:27.132400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a260083-a1c8-4d6b-b8fa-4a7d03be49f5 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:27.133416 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a260083-a1c8-4d6b-b8fa-4a7d03be49f5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:27.133858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a260083-a1c8-4d6b-b8fa-4a7d03be49f5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:27.135428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a260083-a1c8-4d6b-b8fa-4a7d03be49f5 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:27.144673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 186/371] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:27.145845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a260083-a1c8-4d6b-b8fa-4a7d03be49f5 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:27.177602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40f0cc3-9ed1-4c09-8cea-130ee509aa74 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:27.217096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ebb729-2f2f-48b9-b6da-6d47e4094f62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:27.217505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ebb729-2f2f-48b9-b6da-6d47e4094f62 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:27.217794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ebb729-2f2f-48b9-b6da-6d47e4094f62 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:27.218078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ebb729-2f2f-48b9-b6da-6d47e4094f62 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:27.218924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ebb729-2f2f-48b9-b6da-6d47e4094f62 None admin] RBAC: Adding query filter params (group.id=9c84270a862240469756e6d3d3dd760a, role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=b77e955a802a489daa9bd13993979e40) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:27.219239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ebb729-2f2f-48b9-b6da-6d47e4094f62 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:27.220702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ebb729-2f2f-48b9-b6da-6d47e4094f62 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:27.227285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 186/372] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:27 2022] GET /identity/v3/role_assignments?group.id=9c84270a862240469756e6d3d3dd760a&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 260 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:27.232911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ebb729-2f2f-48b9-b6da-6d47e4094f62 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:27.299774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a260083-a1c8-4d6b-b8fa-4a7d03be49f5 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:27.306995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:27.374716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ebb729-2f2f-48b9-b6da-6d47e4094f62 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:27.411714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efca7e4e-1308-4638-99d5-67d8d1bf9bc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:27.412379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efca7e4e-1308-4638-99d5-67d8d1bf9bc9 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:27.412666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efca7e4e-1308-4638-99d5-67d8d1bf9bc9 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:27.412978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efca7e4e-1308-4638-99d5-67d8d1bf9bc9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:27.425614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fc73cd-3ffe-4415-adfc-f29fd3061249 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:27.426289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fc73cd-3ffe-4415-adfc-f29fd3061249 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:27.426581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fc73cd-3ffe-4415-adfc-f29fd3061249 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:27.426866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fc73cd-3ffe-4415-adfc-f29fd3061249 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:27.488060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efca7e4e-1308-4638-99d5-67d8d1bf9bc9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:27.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efca7e4e-1308-4638-99d5-67d8d1bf9bc9 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:27.537754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:27.553134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:27.592932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 187/373] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:27.600596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efca7e4e-1308-4638-99d5-67d8d1bf9bc9 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:27.622271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fc73cd-3ffe-4415-adfc-f29fd3061249 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:27.623820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fc73cd-3ffe-4415-adfc-f29fd3061249 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:27.716788 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efca7e4e-1308-4638-99d5-67d8d1bf9bc9 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:27.752623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aa9cf0-439f-4ec4-94f6-d907d7fdb76e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:27.753683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aa9cf0-439f-4ec4-94f6-d907d7fdb76e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:27.753861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aa9cf0-439f-4ec4-94f6-d907d7fdb76e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:27.754107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aa9cf0-439f-4ec4-94f6-d907d7fdb76e None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:27.755343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aa9cf0-439f-4ec4-94f6-d907d7fdb76e None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:27.756774 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aa9cf0-439f-4ec4-94f6-d907d7fdb76e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:27.758787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 188/374] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:27 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:27.764355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aa9cf0-439f-4ec4-94f6-d907d7fdb76e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:27.785605 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:27.795128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 187/375] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 650 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:27.800353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fc73cd-3ffe-4415-adfc-f29fd3061249 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:27.874310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aa9cf0-439f-4ec4-94f6-d907d7fdb76e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:27.919927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:27.932553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d517d71-6eb3-4913-b5e2-cd709780d0d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:27.933706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d517d71-6eb3-4913-b5e2-cd709780d0d5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:27.934146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d517d71-6eb3-4913-b5e2-cd709780d0d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:27.934563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d517d71-6eb3-4913-b5e2-cd709780d0d5 None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:27.935892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d517d71-6eb3-4913-b5e2-cd709780d0d5 None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:27.938396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d517d71-6eb3-4913-b5e2-cd709780d0d5 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:27.941316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 189/376] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:27 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:27.947611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d517d71-6eb3-4913-b5e2-cd709780d0d5 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:27.983026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fc73cd-3ffe-4415-adfc-f29fd3061249 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:28.044713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84ec490-6571-492d-aa9f-7f67f063cbaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:28.046874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84ec490-6571-492d-aa9f-7f67f063cbaf None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:28.047352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84ec490-6571-492d-aa9f-7f67f063cbaf None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:28.047836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84ec490-6571-492d-aa9f-7f67f063cbaf None admin] PATH_INFO: `/v3/projects/96340edb86a840a88d51d74872a27bac` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:28.049272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84ec490-6571-492d-aa9f-7f67f063cbaf None admin] RBAC: Authorizing `identity:get_project(project_id=96340edb86a840a88d51d74872a27bac)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:28.050913 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84ec490-6571-492d-aa9f-7f67f063cbaf None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:28.053284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 188/377] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:27 2022] GET /identity/v3/projects/96340edb86a840a88d51d74872a27bac => generated 344 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:28.058458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84ec490-6571-492d-aa9f-7f67f063cbaf None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:28.089938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:28.130713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d517d71-6eb3-4913-b5e2-cd709780d0d5 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:28.229928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a99a62-dfa4-4cff-b9dd-598fece720a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:28.231216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a99a62-dfa4-4cff-b9dd-598fece720a1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:28.231701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a99a62-dfa4-4cff-b9dd-598fece720a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:28.232348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a99a62-dfa4-4cff-b9dd-598fece720a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:28.234547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a99a62-dfa4-4cff-b9dd-598fece720a1 None admin] RBAC: Adding query filter params (group.id=9c84270a862240469756e6d3d3dd760a, role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=b77e955a802a489daa9bd13993979e40) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:28.235754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a99a62-dfa4-4cff-b9dd-598fece720a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:28.237458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a99a62-dfa4-4cff-b9dd-598fece720a1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:28.250180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 190/378] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:27 2022] GET /identity/v3/role_assignments?group.id=9c84270a862240469756e6d3d3dd760a&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 260 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:28.252738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84ec490-6571-492d-aa9f-7f67f063cbaf None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:28.295007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32377763-b3ea-418d-b20d-9889516f8159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:28.296213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32377763-b3ea-418d-b20d-9889516f8159 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:28.296701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32377763-b3ea-418d-b20d-9889516f8159 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:28.297214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32377763-b3ea-418d-b20d-9889516f8159 None admin] PATH_INFO: `/v3/groups/50b42b34a078416fb54796cbc20f3ad4` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:28.302574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:28.305557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32377763-b3ea-418d-b20d-9889516f8159 None admin] RBAC: Authorizing `identity:get_group(group_id=50b42b34a078416fb54796cbc20f3ad4)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:28.307959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32377763-b3ea-418d-b20d-9889516f8159 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:28.310205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 189/379] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:28 2022] GET /identity/v3/groups/50b42b34a078416fb54796cbc20f3ad4 => generated 229 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:28.316372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a99a62-dfa4-4cff-b9dd-598fece720a1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:28.465461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a99a62-dfa4-4cff-b9dd-598fece720a1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:28.496667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd351a-0780-49b4-8ffa-358243302c21 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:28.497173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd351a-0780-49b4-8ffa-358243302c21 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:28.497634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cd351a-0780-49b4-8ffa-358243302c21 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:28.500279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 190/380] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:28 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:28.507339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2044c397-1d9b-45e2-9034-96323bd6a388 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:28.507687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2044c397-1d9b-45e2-9034-96323bd6a388 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:28.507892 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2044c397-1d9b-45e2-9034-96323bd6a388 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:28.536425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-2044c397-1d9b-45e2-9034-96323bd6a388 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:28.537658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2044c397-1d9b-45e2-9034-96323bd6a388 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:28.546615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a9a699-bb7b-456b-a2ea-f12ef66ea299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:28.548329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a9a699-bb7b-456b-a2ea-f12ef66ea299 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:28.548612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a9a699-bb7b-456b-a2ea-f12ef66ea299 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:28.548919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a9a699-bb7b-456b-a2ea-f12ef66ea299 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:28.549758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a9a699-bb7b-456b-a2ea-f12ef66ea299 None admin] RBAC: Adding query filter params (group.id=50b42b34a078416fb54796cbc20f3ad4, role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=96340edb86a840a88d51d74872a27bac) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:28.550071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a9a699-bb7b-456b-a2ea-f12ef66ea299 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:28.551534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a9a699-bb7b-456b-a2ea-f12ef66ea299 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:28.559861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 191/381] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:28 2022] GET /identity/v3/role_assignments?group.id=50b42b34a078416fb54796cbc20f3ad4&scope.project.id=96340edb86a840a88d51d74872a27bac&role.id=766799d1b6c049168590571f1a959b3b => generated 260 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:28.640532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 191/382] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Jun 10 12:45:28.662992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a9a699-bb7b-456b-a2ea-f12ef66ea299 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:28.677431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:28.802313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a9a699-bb7b-456b-a2ea-f12ef66ea299 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:28.850283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3793b1f4-abfa-40fe-8ab7-eec0866ac58c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:28.851086 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3793b1f4-abfa-40fe-8ab7-eec0866ac58c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:28.851371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3793b1f4-abfa-40fe-8ab7-eec0866ac58c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:28.851650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3793b1f4-abfa-40fe-8ab7-eec0866ac58c None admin] PATH_INFO: `/v3/roles/member` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:28.870403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3793b1f4-abfa-40fe-8ab7-eec0866ac58c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-3793b1f4-abfa-40fe-8ab7-eec0866ac58c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:28.878584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:28.880006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:28.880868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:28.883012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 192/383] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:28 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:28.888182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2044c397-1d9b-45e2-9034-96323bd6a388 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:29.023871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2044c397-1d9b-45e2-9034-96323bd6a388 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:29.057321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:29.068230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8f644-9f73-49c0-ac06-044b63b4d5c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:29.069012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8f644-9f73-49c0-ac06-044b63b4d5c6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:29.069295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8f644-9f73-49c0-ac06-044b63b4d5c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:29.069568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8f644-9f73-49c0-ac06-044b63b4d5c6 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:29.070341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8f644-9f73-49c0-ac06-044b63b4d5c6 None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:29.070651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8f644-9f73-49c0-ac06-044b63b4d5c6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:29.072378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8f644-9f73-49c0-ac06-044b63b4d5c6 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:29.082545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 192/384] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:28 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:29.090075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3793b1f4-abfa-40fe-8ab7-eec0866ac58c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:29.156353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8de3a74-9e6e-44ce-aa4c-68ab19aba5f3 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:29.157036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8de3a74-9e6e-44ce-aa4c-68ab19aba5f3 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:29.157313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8de3a74-9e6e-44ce-aa4c-68ab19aba5f3 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:29.158679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 193/385] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:29.164725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f2d3ef-9ea9-457b-916f-f7665ec76c21 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:29.165043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f2d3ef-9ea9-457b-916f-f7665ec76c21 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:29.165327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f2d3ef-9ea9-457b-916f-f7665ec76c21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:29.198252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:29.201827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-96f2d3ef-9ea9-457b-916f-f7665ec76c21 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:29.211512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-96f2d3ef-9ea9-457b-916f-f7665ec76c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:29.223637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3793b1f4-abfa-40fe-8ab7-eec0866ac58c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:29.264129 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9d556e-e8de-4a84-b294-9cb3edc1209a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:29.265284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9d556e-e8de-4a84-b294-9cb3edc1209a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:29.265784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9d556e-e8de-4a84-b294-9cb3edc1209a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:29.266291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9d556e-e8de-4a84-b294-9cb3edc1209a None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:29.267689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9d556e-e8de-4a84-b294-9cb3edc1209a None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:29.269327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd9d556e-e8de-4a84-b294-9cb3edc1209a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:29.271506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 193/386] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:29 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:29.277791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9d556e-e8de-4a84-b294-9cb3edc1209a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:29.351715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 194/387] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:29.390314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f2d3ef-9ea9-457b-916f-f7665ec76c21 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:29.390735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd9d556e-e8de-4a84-b294-9cb3edc1209a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:29.426264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6100aa-c659-4fd4-acd0-ee78fc9635ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:29.427156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6100aa-c659-4fd4-acd0-ee78fc9635ef None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:29.427440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6100aa-c659-4fd4-acd0-ee78fc9635ef None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:29.427717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e6100aa-c659-4fd4-acd0-ee78fc9635ef None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:29.428962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6100aa-c659-4fd4-acd0-ee78fc9635ef None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:29.430835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e6100aa-c659-4fd4-acd0-ee78fc9635ef None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:29.433885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 194/388] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:29 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:29.441771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6100aa-c659-4fd4-acd0-ee78fc9635ef None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:29.451617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:29.561001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e6100aa-c659-4fd4-acd0-ee78fc9635ef None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:29.606617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05968819-dc61-4112-b101-d8f2eac34816 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:29.607996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05968819-dc61-4112-b101-d8f2eac34816 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:29.608667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05968819-dc61-4112-b101-d8f2eac34816 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:29.609124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05968819-dc61-4112-b101-d8f2eac34816 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/groups/9c84270a862240469756e6d3d3dd760a/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:29.611314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05968819-dc61-4112-b101-d8f2eac34816 None admin] RBAC: Authorizing `identity:create_grant(project_id=ceb60f2e4cad4cf29d7f7de724549569, group_id=9c84270a862240469756e6d3d3dd760a, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:29.613022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05968819-dc61-4112-b101-d8f2eac34816 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:29.633120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f2d3ef-9ea9-457b-916f-f7665ec76c21 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:29.658749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:29.670741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c2d792-789c-42a8-9c62-4c79ebf6bee6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:29.671754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c2d792-789c-42a8-9c62-4c79ebf6bee6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:29.672237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c2d792-789c-42a8-9c62-4c79ebf6bee6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:29.672669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c2d792-789c-42a8-9c62-4c79ebf6bee6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:29.684178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c2d792-789c-42a8-9c62-4c79ebf6bee6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-80c2d792-789c-42a8-9c62-4c79ebf6bee6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:29.692738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:29.693805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:29.694787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:29.701116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 195/389] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:29 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 311 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:29.702862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c2d792-789c-42a8-9c62-4c79ebf6bee6 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:29.717356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 195/390] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Jun 10 12:45:29 2022] PUT /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/groups/9c84270a862240469756e6d3d3dd760a/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:29.817890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c2d792-789c-42a8-9c62-4c79ebf6bee6 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:29.855298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2f3a74-7b03-4a19-b03f-1782db28d42f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:29.856231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2f3a74-7b03-4a19-b03f-1782db28d42f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:29.856735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2f3a74-7b03-4a19-b03f-1782db28d42f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:29.857182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2f3a74-7b03-4a19-b03f-1782db28d42f None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:29.858094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2f3a74-7b03-4a19-b03f-1782db28d42f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:29.858549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2f3a74-7b03-4a19-b03f-1782db28d42f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:29.860084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2f3a74-7b03-4a19-b03f-1782db28d42f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:29.869748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 196/391] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:29 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:29.875852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05968819-dc61-4112-b101-d8f2eac34816 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:29.948918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:29.996207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05968819-dc61-4112-b101-d8f2eac34816 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:30.037903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebabb8d6-c46a-48ae-9ca1-f158687c162d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:30.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabb8d6-c46a-48ae-9ca1-f158687c162d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:30.039577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabb8d6-c46a-48ae-9ca1-f158687c162d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:30.039997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebabb8d6-c46a-48ae-9ca1-f158687c162d None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:30.041532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebabb8d6-c46a-48ae-9ca1-f158687c162d None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:30.043149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebabb8d6-c46a-48ae-9ca1-f158687c162d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:30.045051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 196/392] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:29 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:30.050726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2f3a74-7b03-4a19-b03f-1782db28d42f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:30.101251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:30.158599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2f3a74-7b03-4a19-b03f-1782db28d42f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:30.193361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213f96e9-cdc5-49b5-ae2f-24bdaa562b79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:30.194279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213f96e9-cdc5-49b5-ae2f-24bdaa562b79 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:30.194567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213f96e9-cdc5-49b5-ae2f-24bdaa562b79 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:30.194843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213f96e9-cdc5-49b5-ae2f-24bdaa562b79 None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:30.196054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213f96e9-cdc5-49b5-ae2f-24bdaa562b79 None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:30.198059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213f96e9-cdc5-49b5-ae2f-24bdaa562b79 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:30.200056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 197/393] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:30 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:30.205761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebabb8d6-c46a-48ae-9ca1-f158687c162d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:30.327945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebabb8d6-c46a-48ae-9ca1-f158687c162d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:30.363660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e06c84b-0545-4cfd-9455-589cad985d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:30.364601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06c84b-0545-4cfd-9455-589cad985d9c None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:30.365022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06c84b-0545-4cfd-9455-589cad985d9c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:30.365310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06c84b-0545-4cfd-9455-589cad985d9c None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/groups/9c84270a862240469756e6d3d3dd760a/roles/df56ac8b9c6f46958342cc16f10cea63` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:30.367538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e06c84b-0545-4cfd-9455-589cad985d9c None admin] RBAC: Authorizing `identity:create_grant(project_id=ceb60f2e4cad4cf29d7f7de724549569, group_id=9c84270a862240469756e6d3d3dd760a, role_id=df56ac8b9c6f46958342cc16f10cea63)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:30.369549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e06c84b-0545-4cfd-9455-589cad985d9c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:30.371847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:30.380116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bdc95-189d-4b4c-9b44-1705ba1afe46 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:30.383521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bdc95-189d-4b4c-9b44-1705ba1afe46 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:30.383521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3bdc95-189d-4b4c-9b44-1705ba1afe46 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:30.383521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 198/394] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:45:30.387377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77160435-ed96-4b35-b7a5-104a68a4f378 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:30.387673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77160435-ed96-4b35-b7a5-104a68a4f378 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:30.387954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77160435-ed96-4b35-b7a5-104a68a4f378 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:30.406535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-77160435-ed96-4b35-b7a5-104a68a4f378 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:30.407737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-77160435-ed96-4b35-b7a5-104a68a4f378 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:30.437247 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 197/395] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Jun 10 12:45:30 2022] PUT /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569/groups/9c84270a862240469756e6d3d3dd760a/roles/df56ac8b9c6f46958342cc16f10cea63 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:30.492575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 199/396] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:30.527738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e06c84b-0545-4cfd-9455-589cad985d9c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:30.658517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e06c84b-0545-4cfd-9455-589cad985d9c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:30.695567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e24c27-15db-44c7-a1fe-3209f7ceeb67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:30.696369 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e24c27-15db-44c7-a1fe-3209f7ceeb67 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:30.696655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e24c27-15db-44c7-a1fe-3209f7ceeb67 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:30.697018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e24c27-15db-44c7-a1fe-3209f7ceeb67 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:30.709892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89e24c27-15db-44c7-a1fe-3209f7ceeb67 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-89e24c27-15db-44c7-a1fe-3209f7ceeb67 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:30.717708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:30.718623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:30.719391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 198/397] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:30.725146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77160435-ed96-4b35-b7a5-104a68a4f378 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:30.773429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:30.837211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77160435-ed96-4b35-b7a5-104a68a4f378 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:30.872355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98feaa66-4e9a-47c4-83d6-5dd4053bb4bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:30.873056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98feaa66-4e9a-47c4-83d6-5dd4053bb4bb None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:30.873604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98feaa66-4e9a-47c4-83d6-5dd4053bb4bb None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:30.873879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98feaa66-4e9a-47c4-83d6-5dd4053bb4bb None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:30.874634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98feaa66-4e9a-47c4-83d6-5dd4053bb4bb None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:30.874942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98feaa66-4e9a-47c4-83d6-5dd4053bb4bb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:30.876337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98feaa66-4e9a-47c4-83d6-5dd4053bb4bb None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:30.885298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 200/398] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:30.890730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e24c27-15db-44c7-a1fe-3209f7ceeb67 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:30.958930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:31.023432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e24c27-15db-44c7-a1fe-3209f7ceeb67 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:31.071535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c25d484-4d0f-4bd6-982d-ac77fdb2dff6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:31.071535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25d484-4d0f-4bd6-982d-ac77fdb2dff6 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.071535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25d484-4d0f-4bd6-982d-ac77fdb2dff6 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.071535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c25d484-4d0f-4bd6-982d-ac77fdb2dff6 None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.072875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25d484-4d0f-4bd6-982d-ac77fdb2dff6 None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:31.074538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c25d484-4d0f-4bd6-982d-ac77fdb2dff6 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:31.076385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 199/399] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:30 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:31.081813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98feaa66-4e9a-47c4-83d6-5dd4053bb4bb None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:31.237314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98feaa66-4e9a-47c4-83d6-5dd4053bb4bb None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:31.245814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:31.252246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d445f59-fd45-44ad-86d2-c3adc6c4916f None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.255888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d445f59-fd45-44ad-86d2-c3adc6c4916f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.255888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d445f59-fd45-44ad-86d2-c3adc6c4916f None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.255888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 200/400] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:31.259633 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1452d249-5134-4444-bdd1-89931c0c92a8 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.260098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1452d249-5134-4444-bdd1-89931c0c92a8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.260389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1452d249-5134-4444-bdd1-89931c0c92a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.282590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bda1b4-988c-4d59-b48d-7612d3d421ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:31.283829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bda1b4-988c-4d59-b48d-7612d3d421ee None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.284325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bda1b4-988c-4d59-b48d-7612d3d421ee None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.284929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bda1b4-988c-4d59-b48d-7612d3d421ee None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.286539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bda1b4-988c-4d59-b48d-7612d3d421ee None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:31.288230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bda1b4-988c-4d59-b48d-7612d3d421ee None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:31.291266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 201/401] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:31 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:31.297813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bda1b4-988c-4d59-b48d-7612d3d421ee None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:31.301198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1452d249-5134-4444-bdd1-89931c0c92a8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:31.304675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1452d249-5134-4444-bdd1-89931c0c92a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:31.403269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 201/402] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:31.405107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578180b7-2e89-4a0c-a230-41e7d8aa0fb2 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.405586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578180b7-2e89-4a0c-a230-41e7d8aa0fb2 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.406038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578180b7-2e89-4a0c-a230-41e7d8aa0fb2 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.407464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 202/403] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:31.424703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bda1b4-988c-4d59-b48d-7612d3d421ee None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:31.429612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea20729-4fe3-4d39-9c7f-1cced161cfe5 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.430126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea20729-4fe3-4d39-9c7f-1cced161cfe5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.430577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea20729-4fe3-4d39-9c7f-1cced161cfe5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.449067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bea20729-4fe3-4d39-9c7f-1cced161cfe5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:31.450566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bea20729-4fe3-4d39-9c7f-1cced161cfe5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:31.465838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b21dd3c-d4f7-4a06-9328-0065b40bcd0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:31.466873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b21dd3c-d4f7-4a06-9328-0065b40bcd0f None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.466983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b21dd3c-d4f7-4a06-9328-0065b40bcd0f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.467257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b21dd3c-d4f7-4a06-9328-0065b40bcd0f None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40/groups/9c84270a862240469756e6d3d3dd760a/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.469227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b21dd3c-d4f7-4a06-9328-0065b40bcd0f None admin] RBAC: Authorizing `identity:create_grant(project_id=b77e955a802a489daa9bd13993979e40, group_id=9c84270a862240469756e6d3d3dd760a, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:31.470662 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b21dd3c-d4f7-4a06-9328-0065b40bcd0f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:31.502297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:31.528797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 203/404] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:31.528797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea20729-4fe3-4d39-9c7f-1cced161cfe5 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:31.544789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 202/405] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Jun 10 12:45:31 2022] PUT /identity/v3/projects/b77e955a802a489daa9bd13993979e40/groups/9c84270a862240469756e6d3d3dd760a/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:31.560909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b21dd3c-d4f7-4a06-9328-0065b40bcd0f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:31.601857 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:31.687555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b21dd3c-d4f7-4a06-9328-0065b40bcd0f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:31.720315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea20729-4fe3-4d39-9c7f-1cced161cfe5 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:31.732996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c615360-d282-402a-9f29-494d20bb55f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:31.734427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c615360-d282-402a-9f29-494d20bb55f6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.735087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c615360-d282-402a-9f29-494d20bb55f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.735658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c615360-d282-402a-9f29-494d20bb55f6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.748693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c615360-d282-402a-9f29-494d20bb55f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-0c615360-d282-402a-9f29-494d20bb55f6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:31.759316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:31.760196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:31.761001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:31.761696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18349b1f-9753-4b43-baa9-b35ca5f35404 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:31.761696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18349b1f-9753-4b43-baa9-b35ca5f35404 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.761696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18349b1f-9753-4b43-baa9-b35ca5f35404 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.761696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18349b1f-9753-4b43-baa9-b35ca5f35404 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.763974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 203/406] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:45:31 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:31.772024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c615360-d282-402a-9f29-494d20bb55f6 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:31.779909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18349b1f-9753-4b43-baa9-b35ca5f35404 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-18349b1f-9753-4b43-baa9-b35ca5f35404 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:31.788589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:31.789516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:31.790287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 204/407] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:31.796128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18349b1f-9753-4b43-baa9-b35ca5f35404 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:31.865232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:31.887513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:31.901375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c615360-d282-402a-9f29-494d20bb55f6 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:31.913150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18349b1f-9753-4b43-baa9-b35ca5f35404 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:31.937832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cf2f19-b68b-4aff-991c-79d76460bbb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:31.939043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cf2f19-b68b-4aff-991c-79d76460bbb9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.939521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cf2f19-b68b-4aff-991c-79d76460bbb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.939993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30cf2f19-b68b-4aff-991c-79d76460bbb9 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.940981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cf2f19-b68b-4aff-991c-79d76460bbb9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:31.941493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cf2f19-b68b-4aff-991c-79d76460bbb9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:31.943178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30cf2f19-b68b-4aff-991c-79d76460bbb9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:31.952511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 204/408] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:31 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:31.958057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cf2f19-b68b-4aff-991c-79d76460bbb9 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:31.993493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6691539d-de25-4ba7-9d22-ab250eedf284 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:31.994798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6691539d-de25-4ba7-9d22-ab250eedf284 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:31.995035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6691539d-de25-4ba7-9d22-ab250eedf284 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:31.995372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6691539d-de25-4ba7-9d22-ab250eedf284 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:31.996268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6691539d-de25-4ba7-9d22-ab250eedf284 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:31.996560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6691539d-de25-4ba7-9d22-ab250eedf284 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:31.998052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6691539d-de25-4ba7-9d22-ab250eedf284 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:32.020087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 205/409] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:32.029933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6691539d-de25-4ba7-9d22-ab250eedf284 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:32.067529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30cf2f19-b68b-4aff-991c-79d76460bbb9 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:32.102081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05298ef0-6cb1-4ea1-8833-4e24ac5ec164 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:32.103171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05298ef0-6cb1-4ea1-8833-4e24ac5ec164 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:32.103551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05298ef0-6cb1-4ea1-8833-4e24ac5ec164 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:32.103923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05298ef0-6cb1-4ea1-8833-4e24ac5ec164 None admin] PATH_INFO: `/v3/groups/50b42b34a078416fb54796cbc20f3ad4` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:32.105256 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05298ef0-6cb1-4ea1-8833-4e24ac5ec164 None admin] RBAC: Authorizing `identity:get_group(group_id=50b42b34a078416fb54796cbc20f3ad4)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:32.106810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05298ef0-6cb1-4ea1-8833-4e24ac5ec164 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:32.108844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 205/410] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:31 2022] GET /identity/v3/groups/50b42b34a078416fb54796cbc20f3ad4 => generated 229 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:32.114268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05298ef0-6cb1-4ea1-8833-4e24ac5ec164 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:32.141220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:32.215188 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05298ef0-6cb1-4ea1-8833-4e24ac5ec164 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:32.252069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab15a41-81c6-475d-8976-b73132f9d261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:32.253346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab15a41-81c6-475d-8976-b73132f9d261 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:32.253934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab15a41-81c6-475d-8976-b73132f9d261 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:32.254321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ab15a41-81c6-475d-8976-b73132f9d261 None admin] PATH_INFO: `/v3/projects/96340edb86a840a88d51d74872a27bac` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:32.255668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab15a41-81c6-475d-8976-b73132f9d261 None admin] RBAC: Authorizing `identity:get_project(project_id=96340edb86a840a88d51d74872a27bac)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:32.257270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ab15a41-81c6-475d-8976-b73132f9d261 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:32.259898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 206/411] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:32 2022] GET /identity/v3/projects/96340edb86a840a88d51d74872a27bac => generated 344 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:32.265530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab15a41-81c6-475d-8976-b73132f9d261 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:32.304012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6691539d-de25-4ba7-9d22-ab250eedf284 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:32.309152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:32.349434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc09b4bf-9be3-4a76-8e7a-230fb2b857ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:32.350690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc09b4bf-9be3-4a76-8e7a-230fb2b857ce None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:32.351105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc09b4bf-9be3-4a76-8e7a-230fb2b857ce None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:32.351388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc09b4bf-9be3-4a76-8e7a-230fb2b857ce None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:32.352559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc09b4bf-9be3-4a76-8e7a-230fb2b857ce None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:32.354562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc09b4bf-9be3-4a76-8e7a-230fb2b857ce None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:32.356574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 206/412] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:32 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:32.367236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc09b4bf-9be3-4a76-8e7a-230fb2b857ce None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:32.431908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:32.442683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ab15a41-81c6-475d-8976-b73132f9d261 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:32.484206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5a48a7-b7fe-4243-bf30-f382b556b659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:32.486798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5a48a7-b7fe-4243-bf30-f382b556b659 None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:32.487083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5a48a7-b7fe-4243-bf30-f382b556b659 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:32.487365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5a48a7-b7fe-4243-bf30-f382b556b659 None admin] PATH_INFO: `/v3/projects/96340edb86a840a88d51d74872a27bac/groups/50b42b34a078416fb54796cbc20f3ad4/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:32.490105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5a48a7-b7fe-4243-bf30-f382b556b659 None admin] RBAC: Authorizing `identity:create_grant(project_id=96340edb86a840a88d51d74872a27bac, group_id=50b42b34a078416fb54796cbc20f3ad4, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:32.491553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5a48a7-b7fe-4243-bf30-f382b556b659 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:32.552065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc09b4bf-9be3-4a76-8e7a-230fb2b857ce None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:32.589329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 207/413] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Jun 10 12:45:32 2022] PUT /identity/v3/projects/96340edb86a840a88d51d74872a27bac/groups/50b42b34a078416fb54796cbc20f3ad4/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:32.593896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb982e1-6d74-4c53-8b92-e02c9b7c3ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:32.594974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb982e1-6d74-4c53-8b92-e02c9b7c3ed8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:32.595264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb982e1-6d74-4c53-8b92-e02c9b7c3ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:32.595545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb982e1-6d74-4c53-8b92-e02c9b7c3ed8 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:32.597005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb982e1-6d74-4c53-8b92-e02c9b7c3ed8 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:32.598645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb982e1-6d74-4c53-8b92-e02c9b7c3ed8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:32.604906 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 207/414] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:32 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:32.607390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5a48a7-b7fe-4243-bf30-f382b556b659 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:32.723129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5a48a7-b7fe-4243-bf30-f382b556b659 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:32.760937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203496e9-b087-47a0-b1f9-3307cac08fa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:32.761849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203496e9-b087-47a0-b1f9-3307cac08fa6 None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:32.762187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203496e9-b087-47a0-b1f9-3307cac08fa6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:32.762470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-203496e9-b087-47a0-b1f9-3307cac08fa6 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40/groups/9c84270a862240469756e6d3d3dd760a/roles/df56ac8b9c6f46958342cc16f10cea63` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:32.764831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203496e9-b087-47a0-b1f9-3307cac08fa6 None admin] RBAC: Authorizing `identity:create_grant(project_id=b77e955a802a489daa9bd13993979e40, group_id=9c84270a862240469756e6d3d3dd760a, role_id=df56ac8b9c6f46958342cc16f10cea63)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:32.766387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-203496e9-b087-47a0-b1f9-3307cac08fa6 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:32.814375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899a735-67f9-4804-b9d6-257f16e9f49f None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:32.815011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899a735-67f9-4804-b9d6-257f16e9f49f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:32.815508 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f899a735-67f9-4804-b9d6-257f16e9f49f None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:32.817396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 208/415] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:32.823403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073bcfa5-d93d-4ad7-9d93-e5241de99074 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:32.824015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073bcfa5-d93d-4ad7-9d93-e5241de99074 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:32.824507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073bcfa5-d93d-4ad7-9d93-e5241de99074 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:32.844168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-073bcfa5-d93d-4ad7-9d93-e5241de99074 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:32.845761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-073bcfa5-d93d-4ad7-9d93-e5241de99074 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:32.853913 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 208/416] 10.222.0.25 () {58 vars in 1442 bytes} [Fri Jun 10 12:45:32 2022] PUT /identity/v3/projects/b77e955a802a489daa9bd13993979e40/groups/9c84270a862240469756e6d3d3dd760a/roles/df56ac8b9c6f46958342cc16f10cea63 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:32.887973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:32.929558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 209/417] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:32.953109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203496e9-b087-47a0-b1f9-3307cac08fa6 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:32.970461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:33.094513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-203496e9-b087-47a0-b1f9-3307cac08fa6 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:33.114538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edc3e86-1a46-42f6-b633-2c96623fc8cf None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:33.115477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edc3e86-1a46-42f6-b633-2c96623fc8cf None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:33.115955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edc3e86-1a46-42f6-b633-2c96623fc8cf None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:33.119954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 210/418] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:33 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:33.125003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab440b3e-7846-4f96-994c-d61350b19e24 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:33.125590 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab440b3e-7846-4f96-994c-d61350b19e24 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:33.126065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab440b3e-7846-4f96-994c-d61350b19e24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:33.129529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44074d8a-ea9e-4bf0-bfec-f864e002ced0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:33.130329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44074d8a-ea9e-4bf0-bfec-f864e002ced0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:33.130610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44074d8a-ea9e-4bf0-bfec-f864e002ced0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:33.130888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44074d8a-ea9e-4bf0-bfec-f864e002ced0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:33.141784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44074d8a-ea9e-4bf0-bfec-f864e002ced0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:33.146861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ab440b3e-7846-4f96-994c-d61350b19e24 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:33.148274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ab440b3e-7846-4f96-994c-d61350b19e24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-44074d8a-ea9e-4bf0-bfec-f864e002ced0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:33.149436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:33.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:33.151182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 209/419] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:32 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:33.156857 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44074d8a-ea9e-4bf0-bfec-f864e002ced0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:33.231061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 211/420] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:33.237138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:33.265928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab440b3e-7846-4f96-994c-d61350b19e24 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:33.279377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44074d8a-ea9e-4bf0-bfec-f864e002ced0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:33.362629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd56a7c4-3b49-4d67-b528-fb163f1fc2a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:33.363207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd56a7c4-3b49-4d67-b528-fb163f1fc2a4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:33.363487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd56a7c4-3b49-4d67-b528-fb163f1fc2a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:33.363757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd56a7c4-3b49-4d67-b528-fb163f1fc2a4 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:33.364508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd56a7c4-3b49-4d67-b528-fb163f1fc2a4 None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:33.364833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd56a7c4-3b49-4d67-b528-fb163f1fc2a4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:33.366245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd56a7c4-3b49-4d67-b528-fb163f1fc2a4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:33.388482 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 210/421] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:33 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:33.394007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd56a7c4-3b49-4d67-b528-fb163f1fc2a4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:33.440448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab440b3e-7846-4f96-994c-d61350b19e24 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:33.453107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:33.506921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3df2c0-d69e-4acf-b5ea-1a25fd7e35e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:33.507732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3df2c0-d69e-4acf-b5ea-1a25fd7e35e2 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:33.508013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3df2c0-d69e-4acf-b5ea-1a25fd7e35e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:33.510728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3df2c0-d69e-4acf-b5ea-1a25fd7e35e2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:33.535581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa3df2c0-d69e-4acf-b5ea-1a25fd7e35e2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-aa3df2c0-d69e-4acf-b5ea-1a25fd7e35e2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:33.544451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:33.545404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:33.546199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 212/422] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:33 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 281 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:33.552072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3df2c0-d69e-4acf-b5ea-1a25fd7e35e2 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:33.602122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd56a7c4-3b49-4d67-b528-fb163f1fc2a4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:33.647646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:33.653468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518ddeaf-9639-4552-a53e-79b9fae26b39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:33.654117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518ddeaf-9639-4552-a53e-79b9fae26b39 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:33.654401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518ddeaf-9639-4552-a53e-79b9fae26b39 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:33.654676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518ddeaf-9639-4552-a53e-79b9fae26b39 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:33.752391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518ddeaf-9639-4552-a53e-79b9fae26b39 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:33.753897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518ddeaf-9639-4552-a53e-79b9fae26b39 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:33.769056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3df2c0-d69e-4acf-b5ea-1a25fd7e35e2 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:33.805538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f85826-e6f1-403f-ac06-ff76308d02b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:33.806779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f85826-e6f1-403f-ac06-ff76308d02b8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:33.807259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f85826-e6f1-403f-ac06-ff76308d02b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:33.807726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f85826-e6f1-403f-ac06-ff76308d02b8 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:33.808745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f85826-e6f1-403f-ac06-ff76308d02b8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:33.809271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f85826-e6f1-403f-ac06-ff76308d02b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:33.811168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f85826-e6f1-403f-ac06-ff76308d02b8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:33.820672 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 213/423] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:33 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:33.834261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f85826-e6f1-403f-ac06-ff76308d02b8 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:33.885132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:33.938318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 211/424] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 544 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jun 10 12:45:33.943906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518ddeaf-9639-4552-a53e-79b9fae26b39 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:34.004914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:34.051528 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518ddeaf-9639-4552-a53e-79b9fae26b39 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:34.088583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6942cf50-7d8c-453e-9266-8d61e9ff892e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:34.089904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6942cf50-7d8c-453e-9266-8d61e9ff892e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:34.090378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6942cf50-7d8c-453e-9266-8d61e9ff892e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:34.090830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6942cf50-7d8c-453e-9266-8d61e9ff892e None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:34.092221 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6942cf50-7d8c-453e-9266-8d61e9ff892e None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:34.093920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6942cf50-7d8c-453e-9266-8d61e9ff892e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:34.096266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 212/425] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:33 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:34.102606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6942cf50-7d8c-453e-9266-8d61e9ff892e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:34.214446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6942cf50-7d8c-453e-9266-8d61e9ff892e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:34.238693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:34.242596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f85826-e6f1-403f-ac06-ff76308d02b8 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:34.255138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d147995a-034d-4467-bb29-80ffb6acb64a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:34.256295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d147995a-034d-4467-bb29-80ffb6acb64a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:34.256887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d147995a-034d-4467-bb29-80ffb6acb64a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:34.257308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d147995a-034d-4467-bb29-80ffb6acb64a None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:34.259307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d147995a-034d-4467-bb29-80ffb6acb64a None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:34.261364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d147995a-034d-4467-bb29-80ffb6acb64a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:34.263530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 213/426] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:34 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:34.269543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d6da9e-93a8-4fce-83fa-d446bf84014e None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:34.269865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d6da9e-93a8-4fce-83fa-d446bf84014e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:34.270082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d6da9e-93a8-4fce-83fa-d446bf84014e None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:34.271508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 214/427] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:34 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:34.272667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d6da9e-93a8-4fce-83fa-d446bf84014e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:34.292721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05e152e-2a93-4883-96dd-b70b76b2c46d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:34.293995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e152e-2a93-4883-96dd-b70b76b2c46d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:34.294476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e152e-2a93-4883-96dd-b70b76b2c46d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:34.294966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05e152e-2a93-4883-96dd-b70b76b2c46d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:34.378057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d6da9e-93a8-4fce-83fa-d446bf84014e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:34.394293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05e152e-2a93-4883-96dd-b70b76b2c46d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:34.396367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b05e152e-2a93-4883-96dd-b70b76b2c46d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:34.414322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05813b3-bdc3-42ff-abef-178c7127b802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:34.415204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05813b3-bdc3-42ff-abef-178c7127b802 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:34.416243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05813b3-bdc3-42ff-abef-178c7127b802 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:34.416719 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05813b3-bdc3-42ff-abef-178c7127b802 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:34.417879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05813b3-bdc3-42ff-abef-178c7127b802 None admin] RBAC: Adding query filter params (group.id=9c84270a862240469756e6d3d3dd760a, role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:34.418534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05813b3-bdc3-42ff-abef-178c7127b802 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:34.420181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05813b3-bdc3-42ff-abef-178c7127b802 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:34.424355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:34.428291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 215/428] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:34 2022] GET /identity/v3/role_assignments?group.id=9c84270a862240469756e6d3d3dd760a&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 611 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:34.429815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf89c40a-88f2-42cc-9061-da2256e283d4 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:34.430315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf89c40a-88f2-42cc-9061-da2256e283d4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:34.430868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf89c40a-88f2-42cc-9061-da2256e283d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:34.457815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-bf89c40a-88f2-42cc-9061-da2256e283d4 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:34.459457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-bf89c40a-88f2-42cc-9061-da2256e283d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:34.518422 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:34.596800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 214/429] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 763 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jun 10 12:45:34.605895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05e152e-2a93-4883-96dd-b70b76b2c46d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:34.619702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 216/430] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:34.656722 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf89c40a-88f2-42cc-9061-da2256e283d4 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:34.718802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05e152e-2a93-4883-96dd-b70b76b2c46d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:34.742112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:34.756555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc264e-9492-477e-a604-10014d41b9fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:34.757406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc264e-9492-477e-a604-10014d41b9fb None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:34.757686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc264e-9492-477e-a604-10014d41b9fb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:34.757960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dc264e-9492-477e-a604-10014d41b9fb None admin] PATH_INFO: `/v3/projects/ceb60f2e4cad4cf29d7f7de724549569` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:34.759218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dc264e-9492-477e-a604-10014d41b9fb None admin] RBAC: Authorizing `identity:get_project(project_id=ceb60f2e4cad4cf29d7f7de724549569)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:34.760706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dc264e-9492-477e-a604-10014d41b9fb None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:34.762430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 215/431] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:34 2022] GET /identity/v3/projects/ceb60f2e4cad4cf29d7f7de724549569 => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:34.767699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc264e-9492-477e-a604-10014d41b9fb None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:34.769003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf89c40a-88f2-42cc-9061-da2256e283d4 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:34.815477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc3b900-1a21-4898-8e06-192ecc35abd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:34.816562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc3b900-1a21-4898-8e06-192ecc35abd6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:34.817031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc3b900-1a21-4898-8e06-192ecc35abd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:34.817514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc3b900-1a21-4898-8e06-192ecc35abd6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:34.833185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc3b900-1a21-4898-8e06-192ecc35abd6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-cfc3b900-1a21-4898-8e06-192ecc35abd6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:34.842295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:34.843173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jun 10 12:45:34.843953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:34.844550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 217/432] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:34 2022] GET /identity/v3/roles/member => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:34.854757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc3b900-1a21-4898-8e06-192ecc35abd6 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:34.885481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dc264e-9492-477e-a604-10014d41b9fb None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:34.886120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:34.946308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c36ab98-9a02-46b5-8f37-f7d4c5b4a855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:34.947214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c36ab98-9a02-46b5-8f37-f7d4c5b4a855 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:34.947496 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c36ab98-9a02-46b5-8f37-f7d4c5b4a855 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:34.947775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c36ab98-9a02-46b5-8f37-f7d4c5b4a855 None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:34.949019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c36ab98-9a02-46b5-8f37-f7d4c5b4a855 None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:34.950594 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c36ab98-9a02-46b5-8f37-f7d4c5b4a855 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:34.952363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 216/433] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:34 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:34.957994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c36ab98-9a02-46b5-8f37-f7d4c5b4a855 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:34.980402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc3b900-1a21-4898-8e06-192ecc35abd6 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:35.017019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dcf7f5-c402-4963-99b9-6cf39adc7894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:35.017704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dcf7f5-c402-4963-99b9-6cf39adc7894 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.017989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dcf7f5-c402-4963-99b9-6cf39adc7894 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.018267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0dcf7f5-c402-4963-99b9-6cf39adc7894 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.019073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dcf7f5-c402-4963-99b9-6cf39adc7894 None admin] RBAC: Adding query filter params (name=member) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:35.019332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dcf7f5-c402-4963-99b9-6cf39adc7894 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:35.020728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0dcf7f5-c402-4963-99b9-6cf39adc7894 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:35.025476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:35.032099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 218/434] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:45:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:35.037441 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dcf7f5-c402-4963-99b9-6cf39adc7894 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:35.072560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c36ab98-9a02-46b5-8f37-f7d4c5b4a855 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:35.116636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-171fc54a-d4cd-422d-bc62-ad4650cc4a27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:35.117342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171fc54a-d4cd-422d-bc62-ad4650cc4a27 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.117654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171fc54a-d4cd-422d-bc62-ad4650cc4a27 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.118007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-171fc54a-d4cd-422d-bc62-ad4650cc4a27 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.118985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171fc54a-d4cd-422d-bc62-ad4650cc4a27 None admin] RBAC: Adding query filter params (group.id=9c84270a862240469756e6d3d3dd760a, role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=ceb60f2e4cad4cf29d7f7de724549569) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:35.119305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171fc54a-d4cd-422d-bc62-ad4650cc4a27 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:35.120780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-171fc54a-d4cd-422d-bc62-ad4650cc4a27 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:35.128947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 217/435] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:34 2022] GET /identity/v3/role_assignments?group.id=9c84270a862240469756e6d3d3dd760a&scope.project.id=ceb60f2e4cad4cf29d7f7de724549569&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 611 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:35.138728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b402b46b-0678-4055-ad1c-d6a8786143c6 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.139027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b402b46b-0678-4055-ad1c-d6a8786143c6 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.139290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b402b46b-0678-4055-ad1c-d6a8786143c6 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.141108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 218/436] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:35.147186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd716f42-433c-4930-b4da-6e179eb52b91 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.147432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd716f42-433c-4930-b4da-6e179eb52b91 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.147710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd716f42-433c-4930-b4da-6e179eb52b91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.149364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:35.171124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0dcf7f5-c402-4963-99b9-6cf39adc7894 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:35.183016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bd716f42-433c-4930-b4da-6e179eb52b91 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:35.184559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bd716f42-433c-4930-b4da-6e179eb52b91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:35.219434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7746ccd-a182-40d3-b5bc-bdbf0d5f2e2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:35.220101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7746ccd-a182-40d3-b5bc-bdbf0d5f2e2f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.220379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7746ccd-a182-40d3-b5bc-bdbf0d5f2e2f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.220657 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7746ccd-a182-40d3-b5bc-bdbf0d5f2e2f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.290072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 219/437] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:35.318128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd716f42-433c-4930-b4da-6e179eb52b91 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:35.347363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7746ccd-a182-40d3-b5bc-bdbf0d5f2e2f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:35.349085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7746ccd-a182-40d3-b5bc-bdbf0d5f2e2f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:35.405154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:35.456995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd716f42-433c-4930-b4da-6e179eb52b91 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:35.459085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:35.480713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 219/438] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:35.482573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e766679-6c00-4f38-b2a5-9b5dbdb2ba38 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.482872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e766679-6c00-4f38-b2a5-9b5dbdb2ba38 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.483418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e766679-6c00-4f38-b2a5-9b5dbdb2ba38 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.484714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 220/439] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:35.487662 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e766679-6c00-4f38-b2a5-9b5dbdb2ba38 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:35.500603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add4f63e-077f-4523-b21a-341fd96afe7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:35.501437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add4f63e-077f-4523-b21a-341fd96afe7d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.501723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add4f63e-077f-4523-b21a-341fd96afe7d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.501998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add4f63e-077f-4523-b21a-341fd96afe7d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.514263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add4f63e-077f-4523-b21a-341fd96afe7d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-add4f63e-077f-4523-b21a-341fd96afe7d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:35.522150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:35.523031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:35.523805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 220/440] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:45:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:35.525258 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64d1b8b-7441-46d6-b75c-386e47042083 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.525937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64d1b8b-7441-46d6-b75c-386e47042083 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.525937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64d1b8b-7441-46d6-b75c-386e47042083 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.542729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-c64d1b8b-7441-46d6-b75c-386e47042083 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:35.543997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c64d1b8b-7441-46d6-b75c-386e47042083 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:35.619877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e766679-6c00-4f38-b2a5-9b5dbdb2ba38 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:35.624642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 221/441] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:35.625585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64d1b8b-7441-46d6-b75c-386e47042083 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:35.658171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbffda7f-ed46-4ab5-af20-5f1ff1099e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:35.659022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbffda7f-ed46-4ab5-af20-5f1ff1099e43 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.659303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbffda7f-ed46-4ab5-af20-5f1ff1099e43 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.659578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbffda7f-ed46-4ab5-af20-5f1ff1099e43 None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.660750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbffda7f-ed46-4ab5-af20-5f1ff1099e43 None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:35.662213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbffda7f-ed46-4ab5-af20-5f1ff1099e43 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:35.668199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 221/442] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:35 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:35.668199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbffda7f-ed46-4ab5-af20-5f1ff1099e43 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:35.670959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:35.714799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:35.725597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64d1b8b-7441-46d6-b75c-386e47042083 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:35.759588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2698b67b-4fd8-4454-891f-0f3ec5969952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:35.759779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbffda7f-ed46-4ab5-af20-5f1ff1099e43 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:35.760253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2698b67b-4fd8-4454-891f-0f3ec5969952 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.760540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2698b67b-4fd8-4454-891f-0f3ec5969952 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.760855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2698b67b-4fd8-4454-891f-0f3ec5969952 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.761605 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2698b67b-4fd8-4454-891f-0f3ec5969952 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:35.761921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2698b67b-4fd8-4454-891f-0f3ec5969952 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:35.763306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2698b67b-4fd8-4454-891f-0f3ec5969952 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:35.771387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 222/443] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:35.772174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2698b67b-4fd8-4454-891f-0f3ec5969952 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:35.792951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e1b177-e5ae-49fd-97d7-358306c2259e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:35.793690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e1b177-e5ae-49fd-97d7-358306c2259e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.793975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e1b177-e5ae-49fd-97d7-358306c2259e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.794253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e1b177-e5ae-49fd-97d7-358306c2259e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.804113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e1b177-e5ae-49fd-97d7-358306c2259e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-82e1b177-e5ae-49fd-97d7-358306c2259e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:35.812576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:35.813491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:35.814277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 222/444] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:35 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:35.814923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e1b177-e5ae-49fd-97d7-358306c2259e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:35.878544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2698b67b-4fd8-4454-891f-0f3ec5969952 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:35.880809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:35.913471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771eb93f-389c-4494-b7a8-8843aa8eb589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:35.914355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771eb93f-389c-4494-b7a8-8843aa8eb589 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.914641 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771eb93f-389c-4494-b7a8-8843aa8eb589 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.914924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771eb93f-389c-4494-b7a8-8843aa8eb589 None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:35.916120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771eb93f-389c-4494-b7a8-8843aa8eb589 None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:35.917603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771eb93f-389c-4494-b7a8-8843aa8eb589 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:35.919220 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 223/445] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:35 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:35.919977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771eb93f-389c-4494-b7a8-8843aa8eb589 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:35.920476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e1b177-e5ae-49fd-97d7-358306c2259e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:35.934922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:35.955142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d881ff18-c1ae-4b65-83cc-b5b5f1c07402 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:35.956517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d881ff18-c1ae-4b65-83cc-b5b5f1c07402 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:35.956830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d881ff18-c1ae-4b65-83cc-b5b5f1c07402 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:35.957113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d881ff18-c1ae-4b65-83cc-b5b5f1c07402 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.018616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771eb93f-389c-4494-b7a8-8843aa8eb589 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:36.022553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d881ff18-c1ae-4b65-83cc-b5b5f1c07402 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.024077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d881ff18-c1ae-4b65-83cc-b5b5f1c07402 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.052118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081b402d-bd82-4e13-b256-c085914a863f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:36.052711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081b402d-bd82-4e13-b256-c085914a863f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:36.053025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081b402d-bd82-4e13-b256-c085914a863f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:36.053309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081b402d-bd82-4e13-b256-c085914a863f None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.054039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081b402d-bd82-4e13-b256-c085914a863f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:36.054350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081b402d-bd82-4e13-b256-c085914a863f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.055712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-081b402d-bd82-4e13-b256-c085914a863f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.063499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 224/446] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:36.064282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081b402d-bd82-4e13-b256-c085914a863f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:36.097959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:36.124555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 223/447] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:36.125419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d881ff18-c1ae-4b65-83cc-b5b5f1c07402 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:36.139491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:36.163717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081b402d-bd82-4e13-b256-c085914a863f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:36.197868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a10be25-490a-4da6-93e7-68354a7bb7d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:36.198627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10be25-490a-4da6-93e7-68354a7bb7d0 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:36.198977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10be25-490a-4da6-93e7-68354a7bb7d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:36.199321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a10be25-490a-4da6-93e7-68354a7bb7d0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.200218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a10be25-490a-4da6-93e7-68354a7bb7d0 None admin] RBAC: Adding query filter params (group.id=9c84270a862240469756e6d3d3dd760a, role.id=dfc4731db42b4f0bbb5e10d84814f9cd, scope.project.id=b77e955a802a489daa9bd13993979e40) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:36.200597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a10be25-490a-4da6-93e7-68354a7bb7d0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.202153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a10be25-490a-4da6-93e7-68354a7bb7d0 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.207846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 225/448] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:36 2022] GET /identity/v3/role_assignments?group.id=9c84270a862240469756e6d3d3dd760a&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=dfc4731db42b4f0bbb5e10d84814f9cd => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:36.208627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a10be25-490a-4da6-93e7-68354a7bb7d0 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:36.222514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d881ff18-c1ae-4b65-83cc-b5b5f1c07402 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:36.266242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73981f9c-418d-4a91-9604-3d8853f15469 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:36.266912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73981f9c-418d-4a91-9604-3d8853f15469 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:36.267243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73981f9c-418d-4a91-9604-3d8853f15469 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:36.271385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73981f9c-418d-4a91-9604-3d8853f15469 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.349593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73981f9c-418d-4a91-9604-3d8853f15469 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.351264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73981f9c-418d-4a91-9604-3d8853f15469 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.365935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:36.369075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:36.372686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a10be25-490a-4da6-93e7-68354a7bb7d0 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:36.414839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1df318-f326-4421-9be2-129244828026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:36.415839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1df318-f326-4421-9be2-129244828026 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:36.416126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1df318-f326-4421-9be2-129244828026 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:36.416414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1df318-f326-4421-9be2-129244828026 None admin] PATH_INFO: `/v3/projects/96340edb86a840a88d51d74872a27bac` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.418302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1df318-f326-4421-9be2-129244828026 None admin] RBAC: Authorizing `identity:get_project(project_id=96340edb86a840a88d51d74872a27bac)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.419759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1df318-f326-4421-9be2-129244828026 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.422449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 226/449] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:36 2022] GET /identity/v3/projects/96340edb86a840a88d51d74872a27bac => generated 344 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:36.427142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1df318-f326-4421-9be2-129244828026 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:36.471925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 224/450] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:36.477575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73981f9c-418d-4a91-9604-3d8853f15469 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:36.547644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1df318-f326-4421-9be2-129244828026 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:36.589171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7c123-c052-4024-8ddb-6ac5c5997bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:36.589948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7c123-c052-4024-8ddb-6ac5c5997bfa None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:36.590237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7c123-c052-4024-8ddb-6ac5c5997bfa None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:36.590519 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa7c123-c052-4024-8ddb-6ac5c5997bfa None admin] PATH_INFO: `/v3/groups/50b42b34a078416fb54796cbc20f3ad4` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.591716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa7c123-c052-4024-8ddb-6ac5c5997bfa None admin] RBAC: Authorizing `identity:get_group(group_id=50b42b34a078416fb54796cbc20f3ad4)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.593220 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa7c123-c052-4024-8ddb-6ac5c5997bfa None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.594491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73981f9c-418d-4a91-9604-3d8853f15469 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:36.594853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 227/451] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:36 2022] GET /identity/v3/groups/50b42b34a078416fb54796cbc20f3ad4 => generated 229 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:36.600459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7c123-c052-4024-8ddb-6ac5c5997bfa None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:36.607984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:36.628366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fbb463-5708-4d54-8ac1-e62b67859b6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:36.629245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fbb463-5708-4d54-8ac1-e62b67859b6b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:36.629527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fbb463-5708-4d54-8ac1-e62b67859b6b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:36.629797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fbb463-5708-4d54-8ac1-e62b67859b6b None admin] PATH_INFO: `/v3/projects/b77e955a802a489daa9bd13993979e40` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.631003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fbb463-5708-4d54-8ac1-e62b67859b6b None admin] RBAC: Authorizing `identity:get_project(project_id=b77e955a802a489daa9bd13993979e40)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.632438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fbb463-5708-4d54-8ac1-e62b67859b6b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.633829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:36.634819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 225/452] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:45:36 2022] GET /identity/v3/projects/b77e955a802a489daa9bd13993979e40 => generated 302 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:36.640107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fbb463-5708-4d54-8ac1-e62b67859b6b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:36.703913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa7c123-c052-4024-8ddb-6ac5c5997bfa None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:36.736435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f39ab1b-58fa-42cf-9670-8f2e78f51200 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:36.736928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fbb463-5708-4d54-8ac1-e62b67859b6b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:36.737104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f39ab1b-58fa-42cf-9670-8f2e78f51200 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:36.737390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f39ab1b-58fa-42cf-9670-8f2e78f51200 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:36.737671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f39ab1b-58fa-42cf-9670-8f2e78f51200 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.738494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f39ab1b-58fa-42cf-9670-8f2e78f51200 None admin] RBAC: Adding query filter params (group.id=50b42b34a078416fb54796cbc20f3ad4, role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=96340edb86a840a88d51d74872a27bac) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:36.738805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f39ab1b-58fa-42cf-9670-8f2e78f51200 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.740241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f39ab1b-58fa-42cf-9670-8f2e78f51200 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.745996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 228/453] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:36 2022] GET /identity/v3/role_assignments?group.id=50b42b34a078416fb54796cbc20f3ad4&scope.project.id=96340edb86a840a88d51d74872a27bac&role.id=766799d1b6c049168590571f1a959b3b => generated 611 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:36.770512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff90aa55-69d4-469b-b532-a3a9818bd965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:36.771380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff90aa55-69d4-469b-b532-a3a9818bd965 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:36.771658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff90aa55-69d4-469b-b532-a3a9818bd965 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:36.771930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff90aa55-69d4-469b-b532-a3a9818bd965 None admin] PATH_INFO: `/v3/groups/9c84270a862240469756e6d3d3dd760a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.773227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff90aa55-69d4-469b-b532-a3a9818bd965 None admin] RBAC: Authorizing `identity:get_group(group_id=9c84270a862240469756e6d3d3dd760a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.774783 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff90aa55-69d4-469b-b532-a3a9818bd965 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.776477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 226/454] 10.222.0.25 () {56 vars in 1180 bytes} [Fri Jun 10 12:45:36 2022] GET /identity/v3/groups/9c84270a862240469756e6d3d3dd760a => generated 226 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:36.782886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f39ab1b-58fa-42cf-9670-8f2e78f51200 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:36.883418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:36.887799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f39ab1b-58fa-42cf-9670-8f2e78f51200 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:36.922602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d9dd15-26b1-425d-b9b9-76ca8707b813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:36.923266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d9dd15-26b1-425d-b9b9-76ca8707b813 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:36.923553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d9dd15-26b1-425d-b9b9-76ca8707b813 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:36.923835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d9dd15-26b1-425d-b9b9-76ca8707b813 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:36.924676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d9dd15-26b1-425d-b9b9-76ca8707b813 None admin] RBAC: Adding query filter params (group.id=9c84270a862240469756e6d3d3dd760a, role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=b77e955a802a489daa9bd13993979e40) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:36.925144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d9dd15-26b1-425d-b9b9-76ca8707b813 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:36.926592 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6d9dd15-26b1-425d-b9b9-76ca8707b813 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:36.932560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 229/455] 10.222.0.25 () {56 vars in 1376 bytes} [Fri Jun 10 12:45:36 2022] GET /identity/v3/role_assignments?group.id=9c84270a862240469756e6d3d3dd760a&scope.project.id=b77e955a802a489daa9bd13993979e40&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 611 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:39.653624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377631ae-3b03-4291-95a3-8022adf29ae8 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:39.654631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377631ae-3b03-4291-95a3-8022adf29ae8 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:39.655113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377631ae-3b03-4291-95a3-8022adf29ae8 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:39.657003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 227/456] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:39.661775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a422dc-9548-4f8c-8396-cf1504893526 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:39.662550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a422dc-9548-4f8c-8396-cf1504893526 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:39.663029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a422dc-9548-4f8c-8396-cf1504893526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:39.689707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-c6a422dc-9548-4f8c-8396-cf1504893526 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:39.691204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c6a422dc-9548-4f8c-8396-cf1504893526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:39.778001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 230/457] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:39.818603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377631ae-3b03-4291-95a3-8022adf29ae8 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:39.893474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:39.948628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad2563-0bf1-445d-b623-3cf4b58ce82f None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:39.949320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad2563-0bf1-445d-b623-3cf4b58ce82f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:39.950019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ad2563-0bf1-445d-b623-3cf4b58ce82f None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:39.951945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 231/458] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:39.957827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93398be-0c5e-4f4d-ad9e-af468ba14901 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:39.958370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93398be-0c5e-4f4d-ad9e-af468ba14901 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:39.958793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93398be-0c5e-4f4d-ad9e-af468ba14901 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:39.968809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377631ae-3b03-4291-95a3-8022adf29ae8 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:39.989812 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-f93398be-0c5e-4f4d-ad9e-af468ba14901 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:39.991387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f93398be-0c5e-4f4d-ad9e-af468ba14901 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:40.012336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe138df-4820-425a-935f-3d8dff1261a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:40.013483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe138df-4820-425a-935f-3d8dff1261a0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.013770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe138df-4820-425a-935f-3d8dff1261a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.014047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe138df-4820-425a-935f-3d8dff1261a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.029396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe138df-4820-425a-935f-3d8dff1261a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:40.030970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe138df-4820-425a-935f-3d8dff1261a0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-bbe138df-4820-425a-935f-3d8dff1261a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:40.055473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:40.056954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:40.057805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:40.057805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:40.057805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:40.057805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:40.060083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 228/459] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:40.061720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead0db9-56f2-4aa1-a6e5-b31038fbe813 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.062233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead0db9-56f2-4aa1-a6e5-b31038fbe813 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.062720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead0db9-56f2-4aa1-a6e5-b31038fbe813 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.064206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 229/460] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:40.069315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dead0db9-56f2-4aa1-a6e5-b31038fbe813 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:40.084867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 232/461] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:40.088561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c4aa43-993f-4c8e-9f49-a45790f8f081 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.089094 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c4aa43-993f-4c8e-9f49-a45790f8f081 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.090242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c4aa43-993f-4c8e-9f49-a45790f8f081 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.115944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-95c4aa43-993f-4c8e-9f49-a45790f8f081 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:40.117853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-95c4aa43-993f-4c8e-9f49-a45790f8f081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:40.141185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:40.188740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:40.193569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dead0db9-56f2-4aa1-a6e5-b31038fbe813 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:40.212859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 233/462] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:40.212859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c4aa43-993f-4c8e-9f49-a45790f8f081 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:40.253221 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5724040-2101-4e90-9648-2a3cd7482217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:40.254180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5724040-2101-4e90-9648-2a3cd7482217 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.255069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5724040-2101-4e90-9648-2a3cd7482217 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.255069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5724040-2101-4e90-9648-2a3cd7482217 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.256077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5724040-2101-4e90-9648-2a3cd7482217 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:40.256592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5724040-2101-4e90-9648-2a3cd7482217 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:40.258388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5724040-2101-4e90-9648-2a3cd7482217 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:40.270773 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 230/463] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:40.271804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5724040-2101-4e90-9648-2a3cd7482217 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:40.336172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c4aa43-993f-4c8e-9f49-a45790f8f081 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:40.382018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3272a6b3-7bfe-4ba1-955f-0c0b3891f953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:40.382866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3272a6b3-7bfe-4ba1-955f-0c0b3891f953 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.383149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3272a6b3-7bfe-4ba1-955f-0c0b3891f953 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.383429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3272a6b3-7bfe-4ba1-955f-0c0b3891f953 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.399521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3272a6b3-7bfe-4ba1-955f-0c0b3891f953 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:40.401065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3272a6b3-7bfe-4ba1-955f-0c0b3891f953 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:40.401701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5724040-2101-4e90-9648-2a3cd7482217 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-3272a6b3-7bfe-4ba1-955f-0c0b3891f953 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:40.426192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:40.427461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:40.427461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:40.427461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:40.427461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:40.427461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 234/464] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:40.427461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3272a6b3-7bfe-4ba1-955f-0c0b3891f953 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:40.437325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:40.445601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d53a1d-ff3e-4484-8c1b-61676cf21e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:40.446929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d53a1d-ff3e-4484-8c1b-61676cf21e54 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.447208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d53a1d-ff3e-4484-8c1b-61676cf21e54 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.449530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d53a1d-ff3e-4484-8c1b-61676cf21e54 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.458421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d53a1d-ff3e-4484-8c1b-61676cf21e54 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:40.460139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d53a1d-ff3e-4484-8c1b-61676cf21e54 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-94d53a1d-ff3e-4484-8c1b-61676cf21e54 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:40.478463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:40.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:40.480288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:40.480288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:40.480288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:40.480288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:40.480288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 231/465] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:40.481445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d53a1d-ff3e-4484-8c1b-61676cf21e54 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:40.497750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:40.545523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3272a6b3-7bfe-4ba1-955f-0c0b3891f953 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:40.580419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeff3d8-65fa-4e3f-8a4e-89227ace757e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:40.581136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeff3d8-65fa-4e3f-8a4e-89227ace757e None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.581422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeff3d8-65fa-4e3f-8a4e-89227ace757e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.581700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aeff3d8-65fa-4e3f-8a4e-89227ace757e None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.582400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeff3d8-65fa-4e3f-8a4e-89227ace757e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:40.583853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aeff3d8-65fa-4e3f-8a4e-89227ace757e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:40.588555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d53a1d-ff3e-4484-8c1b-61676cf21e54 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:40.623159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bcef1d5-5316-46e1-a657-6453e67a0383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:40.623818 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcef1d5-5316-46e1-a657-6453e67a0383 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.624105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcef1d5-5316-46e1-a657-6453e67a0383 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.624386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bcef1d5-5316-46e1-a657-6453e67a0383 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.625142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcef1d5-5316-46e1-a657-6453e67a0383 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:40.625452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcef1d5-5316-46e1-a657-6453e67a0383 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:40.626845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bcef1d5-5316-46e1-a657-6453e67a0383 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:40.638328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 232/466] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:40.639383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3da317-1ca7-4fd1-9955-ced3f88441bb None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3da317-1ca7-4fd1-9955-ced3f88441bb None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.644125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3da317-1ca7-4fd1-9955-ced3f88441bb None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.645091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 233/467] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:40 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:40.645856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3da317-1ca7-4fd1-9955-ced3f88441bb None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:40.668833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 235/468] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:45:40 2022] POST /identity/v3/users => generated 246 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:45:40.669645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeff3d8-65fa-4e3f-8a4e-89227ace757e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:40.749603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:40.770120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:40.787826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3da317-1ca7-4fd1-9955-ced3f88441bb None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:40.827356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a28d6a32-c13b-4605-8608-f76d38f864d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:40.828218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28d6a32-c13b-4605-8608-f76d38f864d7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.828431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28d6a32-c13b-4605-8608-f76d38f864d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.828703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a28d6a32-c13b-4605-8608-f76d38f864d7 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.829538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28d6a32-c13b-4605-8608-f76d38f864d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:40.829762 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28d6a32-c13b-4605-8608-f76d38f864d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:40.831171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a28d6a32-c13b-4605-8608-f76d38f864d7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:40.833154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aeff3d8-65fa-4e3f-8a4e-89227ace757e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:40.850185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 234/469] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:40.850185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7a9a37-8562-4481-acbc-19cac4b10988 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.850185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7a9a37-8562-4481-acbc-19cac4b10988 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.850185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7a9a37-8562-4481-acbc-19cac4b10988 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.868807 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ed7a9a37-8562-4481-acbc-19cac4b10988 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:40.869984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ed7a9a37-8562-4481-acbc-19cac4b10988 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:40.878960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b107c9-7cb3-4e0b-976d-f1c167560b5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:40.879623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b107c9-7cb3-4e0b-976d-f1c167560b5f None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:40.879906 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b107c9-7cb3-4e0b-976d-f1c167560b5f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:40.880399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b107c9-7cb3-4e0b-976d-f1c167560b5f None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:40.881545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b107c9-7cb3-4e0b-976d-f1c167560b5f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:40.882360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b107c9-7cb3-4e0b-976d-f1c167560b5f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:40.943217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 235/470] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:40.944117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7a9a37-8562-4481-acbc-19cac4b10988 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:40.958738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 236/471] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:45:40 2022] POST /identity/v3/users => generated 248 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:45:40.978301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b107c9-7cb3-4e0b-976d-f1c167560b5f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:41.031275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:41.088102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7a9a37-8562-4481-acbc-19cac4b10988 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:41.107617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b107c9-7cb3-4e0b-976d-f1c167560b5f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:41.128742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:41.139562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f23e57b-5674-408d-a6b3-2c29eb3a6b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:41.139562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f23e57b-5674-408d-a6b3-2c29eb3a6b17 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:41.139562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f23e57b-5674-408d-a6b3-2c29eb3a6b17 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:41.139562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f23e57b-5674-408d-a6b3-2c29eb3a6b17 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:41.139917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f23e57b-5674-408d-a6b3-2c29eb3a6b17 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:41.142051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f23e57b-5674-408d-a6b3-2c29eb3a6b17 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:41.154356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983fa704-55ff-417d-b39d-8756a2a097e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:41.155207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983fa704-55ff-417d-b39d-8756a2a097e3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:41.155704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983fa704-55ff-417d-b39d-8756a2a097e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:41.156156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983fa704-55ff-417d-b39d-8756a2a097e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:41.168230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983fa704-55ff-417d-b39d-8756a2a097e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:41.169813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983fa704-55ff-417d-b39d-8756a2a097e3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-983fa704-55ff-417d-b39d-8756a2a097e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:41.187417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:41.188357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:41.189235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:41.189235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:41.189235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:41.189235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:41.189235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 237/472] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:41.190097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b8535d-b73f-440a-b8f2-3aa8fcb4b58c None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:41.190389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b8535d-b73f-440a-b8f2-3aa8fcb4b58c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:41.190663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b8535d-b73f-440a-b8f2-3aa8fcb4b58c None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:41.192204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 238/473] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:41.194766 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b8535d-b73f-440a-b8f2-3aa8fcb4b58c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:41.226861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 236/474] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:45:40 2022] POST /identity/v3/users => generated 248 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:45:41.234896 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c428da4e-05d7-40f1-bf56-85fae0ba3b1d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:41.235549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c428da4e-05d7-40f1-bf56-85fae0ba3b1d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:41.235826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c428da4e-05d7-40f1-bf56-85fae0ba3b1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:41.270769 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c428da4e-05d7-40f1-bf56-85fae0ba3b1d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:41.275293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c428da4e-05d7-40f1-bf56-85fae0ba3b1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:41.353052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b8535d-b73f-440a-b8f2-3aa8fcb4b58c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:41.393415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2a4159-50ba-4b76-a645-fde1c841b5ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:41.394889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2a4159-50ba-4b76-a645-fde1c841b5ef None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:41.395403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2a4159-50ba-4b76-a645-fde1c841b5ef None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:41.395885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2a4159-50ba-4b76-a645-fde1c841b5ef None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:41.396866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2a4159-50ba-4b76-a645-fde1c841b5ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:41.397541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2a4159-50ba-4b76-a645-fde1c841b5ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:41.399839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f2a4159-50ba-4b76-a645-fde1c841b5ef None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:41.420462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 239/475] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:41.430021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2a4159-50ba-4b76-a645-fde1c841b5ef None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:41.449113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 237/476] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:41.466983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:41.480289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c428da4e-05d7-40f1-bf56-85fae0ba3b1d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:41.534584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:41.544505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2a4159-50ba-4b76-a645-fde1c841b5ef None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:41.580884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71811ca4-ec2a-47ba-b091-2a303195b6de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:41.581659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71811ca4-ec2a-47ba-b091-2a303195b6de None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:41.581996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71811ca4-ec2a-47ba-b091-2a303195b6de None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:41.582331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71811ca4-ec2a-47ba-b091-2a303195b6de None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:41.583095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71811ca4-ec2a-47ba-b091-2a303195b6de None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:41.584599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71811ca4-ec2a-47ba-b091-2a303195b6de None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:41.592255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c428da4e-05d7-40f1-bf56-85fae0ba3b1d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:41.631182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:41.632090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:41.632469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:41.632694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:41.633479 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:41.634931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:41.675360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 240/477] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:45:41 2022] POST /identity/v3/users => generated 249 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:45:41.693451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.sql.core [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jun 10 12:45:41.693451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jun 10 12:45:41.693451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [parameters: {'id': '5b6196b397484668a307749a307c7046', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jun 10 12:45:41.693451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80300) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.dialect.do_execute( Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application result = self._query(query) Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application conn.query(q) Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application result.read() Jun 10 12:45:41.716750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application packet.raise_for_error() Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return ref.to_dict() Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jun 10 12:45:41.717693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application next(self.gen) Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application yield resource Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application next(self.gen) Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.session.rollback() Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.force_reraise() Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise self.value Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application session.commit() Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jun 10 12:45:41.718701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._prepare_impl() Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.session.flush() Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._flush(objects) Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application compat.raise_( Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application flush_context.execute() Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rec.execute(self) Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jun 10 12:45:41.720559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application _emit_insert_statements( Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application c = connection._execute_20( Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ret = self._execute_context( Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jun 10 12:45:41.722628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.dialect.do_execute( Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application result = self._query(query) Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application conn.query(q) Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application result.read() Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jun 10 12:45:41.724886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application packet.raise_for_error() Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [parameters: {'id': '5b6196b397484668a307749a307c7046', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:41.731030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jun 10 12:45:41.733081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:41.735042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 238/478] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Jun 10 12:45:41 2022] POST /identity/v3/roles => generated 144 bytes in 239 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:41.735173 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71811ca4-ec2a-47ba-b091-2a303195b6de None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:41.843888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:41.867551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71811ca4-ec2a-47ba-b091-2a303195b6de None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:41.904333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120be08f-fb24-44f1-973e-76fd3acbba16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:41.905185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120be08f-fb24-44f1-973e-76fd3acbba16 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:41.905465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120be08f-fb24-44f1-973e-76fd3acbba16 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:41.905736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120be08f-fb24-44f1-973e-76fd3acbba16 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:41.917786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120be08f-fb24-44f1-973e-76fd3acbba16 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-120be08f-fb24-44f1-973e-76fd3acbba16 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:41.925904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:41.926767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:41.927555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 241/479] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:45:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:41.934156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:42.043294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b66fc4-8dd0-4e70-b9a4-18063b7ab337 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:42.059609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:42.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408ea886-59a7-4998-9abd-042950c64559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:42.078681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408ea886-59a7-4998-9abd-042950c64559 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:42.078966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408ea886-59a7-4998-9abd-042950c64559 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:42.079243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408ea886-59a7-4998-9abd-042950c64559 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:42.080025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408ea886-59a7-4998-9abd-042950c64559 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:42.080431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408ea886-59a7-4998-9abd-042950c64559 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:42.081915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408ea886-59a7-4998-9abd-042950c64559 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:42.097066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 239/480] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:43.057531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dbebc3-ee0d-4616-a2b4-a52eb168b5d9 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:43.058034 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dbebc3-ee0d-4616-a2b4-a52eb168b5d9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:43.058107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28dbebc3-ee0d-4616-a2b4-a52eb168b5d9 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:43.059682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 242/481] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:43.064993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd31be2-c2eb-4b20-a77b-c612198ce452 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:43.065397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd31be2-c2eb-4b20-a77b-c612198ce452 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:43.065559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd31be2-c2eb-4b20-a77b-c612198ce452 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:43.082503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5dd31be2-c2eb-4b20-a77b-c612198ce452 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:43.083681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5dd31be2-c2eb-4b20-a77b-c612198ce452 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:43.155294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 240/482] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:43.177936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dbebc3-ee0d-4616-a2b4-a52eb168b5d9 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:43.280455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28dbebc3-ee0d-4616-a2b4-a52eb168b5d9 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:43.314105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d539ea3-f62e-49e5-b028-733e87712f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:43.314820 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d539ea3-f62e-49e5-b028-733e87712f8c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:43.315103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d539ea3-f62e-49e5-b028-733e87712f8c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:43.315377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d539ea3-f62e-49e5-b028-733e87712f8c None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:43.331617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d539ea3-f62e-49e5-b028-733e87712f8c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-6d539ea3-f62e-49e5-b028-733e87712f8c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:43.340042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:43.340954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:43.341746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:43.350138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd31be2-c2eb-4b20-a77b-c612198ce452 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:43.352150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 243/483] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:43.500678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd31be2-c2eb-4b20-a77b-c612198ce452 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:43.505751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:43.544403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800a56b9-df2d-42dc-a260-7de5a3a017fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:43.545194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800a56b9-df2d-42dc-a260-7de5a3a017fa None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:43.545489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800a56b9-df2d-42dc-a260-7de5a3a017fa None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:43.545772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800a56b9-df2d-42dc-a260-7de5a3a017fa None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:43.546546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800a56b9-df2d-42dc-a260-7de5a3a017fa None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:43.546856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800a56b9-df2d-42dc-a260-7de5a3a017fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:43.548235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800a56b9-df2d-42dc-a260-7de5a3a017fa None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:43.559969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 241/484] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:43 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:43.565692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d539ea3-f62e-49e5-b028-733e87712f8c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:43.577933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:43.736463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d539ea3-f62e-49e5-b028-733e87712f8c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:43.751927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d110ec-f324-4df4-b957-c363904b0beb None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:43.752217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d110ec-f324-4df4-b957-c363904b0beb None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:43.752498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d110ec-f324-4df4-b957-c363904b0beb None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:43.753900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 242/485] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:43.761647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14c0978-3965-4f81-ae30-0863e6d1329e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:43.761935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14c0978-3965-4f81-ae30-0863e6d1329e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:43.762212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14c0978-3965-4f81-ae30-0863e6d1329e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:43.790418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcfbd10-1491-413d-9f5a-9127cac58526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:43.791290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-d14c0978-3965-4f81-ae30-0863e6d1329e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:43.792476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d14c0978-3965-4f81-ae30-0863e6d1329e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:43.795127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcfbd10-1491-413d-9f5a-9127cac58526 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:43.795414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcfbd10-1491-413d-9f5a-9127cac58526 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:43.795693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcfbd10-1491-413d-9f5a-9127cac58526 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:43.806857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcfbd10-1491-413d-9f5a-9127cac58526 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:43.808347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcfbd10-1491-413d-9f5a-9127cac58526 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-7fcfbd10-1491-413d-9f5a-9127cac58526 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:43.834904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:43.835745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:43.837390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:43.837390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:43.837390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:43.837390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:43.837390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 244/486] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:43.845944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcfbd10-1491-413d-9f5a-9127cac58526 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:43.919610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 243/487] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:43.950513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:43.966497 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14c0978-3965-4f81-ae30-0863e6d1329e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:44.047083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcfbd10-1491-413d-9f5a-9127cac58526 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:44.109964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84fce27-addc-46f4-b167-dc1ed2f7c606 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:44.110568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:44.111524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84fce27-addc-46f4-b167-dc1ed2f7c606 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.111892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84fce27-addc-46f4-b167-dc1ed2f7c606 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.112170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84fce27-addc-46f4-b167-dc1ed2f7c606 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.112924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84fce27-addc-46f4-b167-dc1ed2f7c606 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:44.113234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84fce27-addc-46f4-b167-dc1ed2f7c606 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:44.114615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a84fce27-addc-46f4-b167-dc1ed2f7c606 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:44.117338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14c0978-3965-4f81-ae30-0863e6d1329e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:44.127310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 245/488] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:44.128611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4084f58f-fa1f-406a-baea-3fa43f57f58a None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.128932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4084f58f-fa1f-406a-baea-3fa43f57f58a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.129302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4084f58f-fa1f-406a-baea-3fa43f57f58a None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.130778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 246/489] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:44.132482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4084f58f-fa1f-406a-baea-3fa43f57f58a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:44.165900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25f191d3-f81b-4de6-8d0d-12807b42708f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:44.167018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f191d3-f81b-4de6-8d0d-12807b42708f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.168010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f191d3-f81b-4de6-8d0d-12807b42708f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.168520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25f191d3-f81b-4de6-8d0d-12807b42708f None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.183989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25f191d3-f81b-4de6-8d0d-12807b42708f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-25f191d3-f81b-4de6-8d0d-12807b42708f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:44.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:44.194443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:44.195220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:44.198240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 244/490] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:44.200177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c326a9-4f2f-4ea7-a888-4839c0ac2b04 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.200617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c326a9-4f2f-4ea7-a888-4839c0ac2b04 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.201678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c326a9-4f2f-4ea7-a888-4839c0ac2b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.222405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-70c326a9-4f2f-4ea7-a888-4839c0ac2b04 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:44.224023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-70c326a9-4f2f-4ea7-a888-4839c0ac2b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:44.262689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4084f58f-fa1f-406a-baea-3fa43f57f58a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:44.285265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:44.308942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 245/491] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:44.308942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c326a9-4f2f-4ea7-a888-4839c0ac2b04 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:44.319262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c6b47d-a1ea-424e-a4da-e021953c861b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:44.323232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c6b47d-a1ea-424e-a4da-e021953c861b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.323232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c6b47d-a1ea-424e-a4da-e021953c861b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.323232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c6b47d-a1ea-424e-a4da-e021953c861b None admin] PATH_INFO: `/v3/users/nova` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.340266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c6b47d-a1ea-424e-a4da-e021953c861b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:44.340266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c6b47d-a1ea-424e-a4da-e021953c861b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-d4c6b47d-a1ea-424e-a4da-e021953c861b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:44.358629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:44.359613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:44.360565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 247/492] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Jun 10 12:45:44 2022] GET /identity/v3/users/nova => generated 82 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:44.365675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c6b47d-a1ea-424e-a4da-e021953c861b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:44.420886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:44.440611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c326a9-4f2f-4ea7-a888-4839c0ac2b04 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:44.478598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c6b47d-a1ea-424e-a4da-e021953c861b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:44.479727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571f24dc-fdd3-4d6b-add4-11ae8bf41a1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:44.480636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571f24dc-fdd3-4d6b-add4-11ae8bf41a1c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.481122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571f24dc-fdd3-4d6b-add4-11ae8bf41a1c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.481537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571f24dc-fdd3-4d6b-add4-11ae8bf41a1c None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.482421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571f24dc-fdd3-4d6b-add4-11ae8bf41a1c None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:44.482886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571f24dc-fdd3-4d6b-add4-11ae8bf41a1c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:44.484459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571f24dc-fdd3-4d6b-add4-11ae8bf41a1c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:44.493944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 246/493] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:44.494942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571f24dc-fdd3-4d6b-add4-11ae8bf41a1c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:44.515115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbdf66a-948a-4f37-bfa6-6500482c0d96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:44.515912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbdf66a-948a-4f37-bfa6-6500482c0d96 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.516320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbdf66a-948a-4f37-bfa6-6500482c0d96 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.516601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbdf66a-948a-4f37-bfa6-6500482c0d96 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.528668 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbdf66a-948a-4f37-bfa6-6500482c0d96 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-5dbdf66a-948a-4f37-bfa6-6500482c0d96 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:44.536800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:44.537692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:44.538582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 248/494] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:44.539801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525f4d13-a4fc-4348-859d-323994c4483b None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.539801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525f4d13-a4fc-4348-859d-323994c4483b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.539953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525f4d13-a4fc-4348-859d-323994c4483b None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.544802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 249/495] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:44.544802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525f4d13-a4fc-4348-859d-323994c4483b None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:44.583462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:44.606764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571f24dc-fdd3-4d6b-add4-11ae8bf41a1c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:44.643646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d875c326-70a3-469a-9059-6515d970725b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:44.644519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d875c326-70a3-469a-9059-6515d970725b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.645003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d875c326-70a3-469a-9059-6515d970725b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.645285 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d875c326-70a3-469a-9059-6515d970725b None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.646128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d875c326-70a3-469a-9059-6515d970725b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:44.646439 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d875c326-70a3-469a-9059-6515d970725b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:44.648043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d875c326-70a3-469a-9059-6515d970725b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:44.651728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525f4d13-a4fc-4348-859d-323994c4483b None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:44.689858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 247/496] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Jun 10 12:45:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:44.689858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d875c326-70a3-469a-9059-6515d970725b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:44.691061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9111616-ea20-41b4-83f8-6a93fc7d6b9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:44.691924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9111616-ea20-41b4-83f8-6a93fc7d6b9c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.692190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9111616-ea20-41b4-83f8-6a93fc7d6b9c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.692468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9111616-ea20-41b4-83f8-6a93fc7d6b9c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.699012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:44.705011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9111616-ea20-41b4-83f8-6a93fc7d6b9c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:44.706533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9111616-ea20-41b4-83f8-6a93fc7d6b9c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-f9111616-ea20-41b4-83f8-6a93fc7d6b9c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:44.725451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:44.726337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:44.727207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:44.727207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:44.727207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:44.727207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:44.727207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 250/497] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:44.727941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbff938-14bb-436c-9ff7-bf39d73c4fb5 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.728274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbff938-14bb-436c-9ff7-bf39d73c4fb5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.728451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bbff938-14bb-436c-9ff7-bf39d73c4fb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.747640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-7bbff938-14bb-436c-9ff7-bf39d73c4fb5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:44.749204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7bbff938-14bb-436c-9ff7-bf39d73c4fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:44.806125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d875c326-70a3-469a-9059-6515d970725b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:44.820260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 251/498] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:44.820969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbff938-14bb-436c-9ff7-bf39d73c4fb5 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:44.840504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:44.857599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038bd8c4-736b-4475-a60f-002160fa2c92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:44.858289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038bd8c4-736b-4475-a60f-002160fa2c92 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.858572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038bd8c4-736b-4475-a60f-002160fa2c92 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.858848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038bd8c4-736b-4475-a60f-002160fa2c92 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.859597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038bd8c4-736b-4475-a60f-002160fa2c92 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:44.859900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038bd8c4-736b-4475-a60f-002160fa2c92 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:44.861339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038bd8c4-736b-4475-a60f-002160fa2c92 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:44.870421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 248/499] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:44.871075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038bd8c4-736b-4475-a60f-002160fa2c92 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:44.933724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bbff938-14bb-436c-9ff7-bf39d73c4fb5 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:44.940061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:44.976365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771f1a22-f930-4ada-a251-4a30708ae52c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:44.977050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771f1a22-f930-4ada-a251-4a30708ae52c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:44.977914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771f1a22-f930-4ada-a251-4a30708ae52c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:44.978200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771f1a22-f930-4ada-a251-4a30708ae52c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:44.980040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038bd8c4-736b-4475-a60f-002160fa2c92 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.017633 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f684e-6625-4fa4-b247-dd78eafea3ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.018284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f684e-6625-4fa4-b247-dd78eafea3ab None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.018569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f684e-6625-4fa4-b247-dd78eafea3ab None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.018844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f684e-6625-4fa4-b247-dd78eafea3ab None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.019561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5f684e-6625-4fa4-b247-dd78eafea3ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:45.019874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5f684e-6625-4fa4-b247-dd78eafea3ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.021332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5f684e-6625-4fa4-b247-dd78eafea3ab None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.033326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 249/500] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:45.034117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f684e-6625-4fa4-b247-dd78eafea3ab None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.052065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771f1a22-f930-4ada-a251-4a30708ae52c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.054526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-771f1a22-f930-4ada-a251-4a30708ae52c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.071436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:45.145900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f684e-6625-4fa4-b247-dd78eafea3ab None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.165225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 252/501] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:45.166076 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771f1a22-f930-4ada-a251-4a30708ae52c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.173043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:45.181214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800c7406-ca2a-4bb6-8d77-ced12f2a2465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.181943 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800c7406-ca2a-4bb6-8d77-ced12f2a2465 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.188173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800c7406-ca2a-4bb6-8d77-ced12f2a2465 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.188173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800c7406-ca2a-4bb6-8d77-ced12f2a2465 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.193125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800c7406-ca2a-4bb6-8d77-ced12f2a2465 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-800c7406-ca2a-4bb6-8d77-ced12f2a2465 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:45.201026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:45.201988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:45.202766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 250/502] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:45.203864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6774bd6a-01e1-4ada-b299-9495f089bf53 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.203937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6774bd6a-01e1-4ada-b299-9495f089bf53 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.204200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6774bd6a-01e1-4ada-b299-9495f089bf53 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.207899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 251/503] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:45.207899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6774bd6a-01e1-4ada-b299-9495f089bf53 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.264080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771f1a22-f930-4ada-a251-4a30708ae52c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.281289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:45.298384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3760add-c455-4bc6-8e92-538c0050dd4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.299207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3760add-c455-4bc6-8e92-538c0050dd4c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.299436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3760add-c455-4bc6-8e92-538c0050dd4c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.299707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3760add-c455-4bc6-8e92-538c0050dd4c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.306163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6774bd6a-01e1-4ada-b299-9495f089bf53 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.309890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3760add-c455-4bc6-8e92-538c0050dd4c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.311340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3760add-c455-4bc6-8e92-538c0050dd4c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-a3760add-c455-4bc6-8e92-538c0050dd4c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:45.328889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:45.329918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:45.330943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:45.330943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:45.330943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:45.330943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:45.330943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 253/504] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:45.331250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3760add-c455-4bc6-8e92-538c0050dd4c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.347642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63891463-959e-4e7e-afee-899162d3075f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.348424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63891463-959e-4e7e-afee-899162d3075f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.348703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63891463-959e-4e7e-afee-899162d3075f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.349002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63891463-959e-4e7e-afee-899162d3075f None admin] PATH_INFO: `/v3/users/glance` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.355955 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63891463-959e-4e7e-afee-899162d3075f None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.357455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63891463-959e-4e7e-afee-899162d3075f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-63891463-959e-4e7e-afee-899162d3075f None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:45.373208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:45.374245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 252/505] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/users/glance => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:45.375147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63891463-959e-4e7e-afee-899162d3075f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.422644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:45.431538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3760add-c455-4bc6-8e92-538c0050dd4c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.473785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63891463-959e-4e7e-afee-899162d3075f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.478107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22942a07-1068-4279-bb6c-45898cd2c056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.478903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22942a07-1068-4279-bb6c-45898cd2c056 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.479306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22942a07-1068-4279-bb6c-45898cd2c056 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.479770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22942a07-1068-4279-bb6c-45898cd2c056 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.490529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22942a07-1068-4279-bb6c-45898cd2c056 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.492087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22942a07-1068-4279-bb6c-45898cd2c056 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.510014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aee4d81-2089-4e84-a975-8432098baa48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.510668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aee4d81-2089-4e84-a975-8432098baa48 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-22942a07-1068-4279-bb6c-45898cd2c056 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:45.510781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:45.511548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:45.512379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:45.512379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:45.512379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:45.512379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:45.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aee4d81-2089-4e84-a975-8432098baa48 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aee4d81-2089-4e84-a975-8432098baa48 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.512590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aee4d81-2089-4e84-a975-8432098baa48 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:45.513126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 254/506] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:45.513212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aee4d81-2089-4e84-a975-8432098baa48 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.513459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc216884-949b-49c0-bf25-b72317bfca07 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.513614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aee4d81-2089-4e84-a975-8432098baa48 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.513750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc216884-949b-49c0-bf25-b72317bfca07 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.514031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc216884-949b-49c0-bf25-b72317bfca07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.528126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 253/507] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:45.528126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aee4d81-2089-4e84-a975-8432098baa48 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.531140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bc216884-949b-49c0-bf25-b72317bfca07 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:45.532357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bc216884-949b-49c0-bf25-b72317bfca07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:45.549365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:45.597399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 255/508] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:45.598235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc216884-949b-49c0-bf25-b72317bfca07 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.636874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aee4d81-2089-4e84-a975-8432098baa48 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.647142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:45.669906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25df846a-4191-418a-a7ec-9fe7f98bdad0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.670562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25df846a-4191-418a-a7ec-9fe7f98bdad0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.670852 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25df846a-4191-418a-a7ec-9fe7f98bdad0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.671119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25df846a-4191-418a-a7ec-9fe7f98bdad0 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.671828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25df846a-4191-418a-a7ec-9fe7f98bdad0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:45.672138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25df846a-4191-418a-a7ec-9fe7f98bdad0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.673545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25df846a-4191-418a-a7ec-9fe7f98bdad0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.686304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 254/509] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:45.688579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25df846a-4191-418a-a7ec-9fe7f98bdad0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.696538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc216884-949b-49c0-bf25-b72317bfca07 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.730705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9681c292-96d1-4768-b27b-7839f1034667 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.731300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9681c292-96d1-4768-b27b-7839f1034667 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.731584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9681c292-96d1-4768-b27b-7839f1034667 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.731936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9681c292-96d1-4768-b27b-7839f1034667 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.732689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9681c292-96d1-4768-b27b-7839f1034667 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:45.733022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9681c292-96d1-4768-b27b-7839f1034667 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.734458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9681c292-96d1-4768-b27b-7839f1034667 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.769968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 256/510] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:45.770799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9681c292-96d1-4768-b27b-7839f1034667 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.783926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:45.797118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25df846a-4191-418a-a7ec-9fe7f98bdad0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.831338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9346e5e2-61f6-4e70-a58e-b50af4802f5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.831983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9346e5e2-61f6-4e70-a58e-b50af4802f5b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.832264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9346e5e2-61f6-4e70-a58e-b50af4802f5b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.832538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9346e5e2-61f6-4e70-a58e-b50af4802f5b None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.833291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9346e5e2-61f6-4e70-a58e-b50af4802f5b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:45.833592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9346e5e2-61f6-4e70-a58e-b50af4802f5b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.834971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9346e5e2-61f6-4e70-a58e-b50af4802f5b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.846104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 255/511] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:45.846882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9346e5e2-61f6-4e70-a58e-b50af4802f5b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.872709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9681c292-96d1-4768-b27b-7839f1034667 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.877010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:45.907220 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b688bf-3f3a-47b1-95e5-9356f5132780 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.908005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b688bf-3f3a-47b1-95e5-9356f5132780 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.908288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b688bf-3f3a-47b1-95e5-9356f5132780 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.908561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b688bf-3f3a-47b1-95e5-9356f5132780 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.918318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b688bf-3f3a-47b1-95e5-9356f5132780 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.919792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b688bf-3f3a-47b1-95e5-9356f5132780 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-e4b688bf-3f3a-47b1-95e5-9356f5132780 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:45.937500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:45.938331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:45.939206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:45.939206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:45.939206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:45.939206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:45.939206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 257/512] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:45.939971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b688bf-3f3a-47b1-95e5-9356f5132780 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:45.949467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9346e5e2-61f6-4e70-a58e-b50af4802f5b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:45.983277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532d3984-94cc-4756-8758-5e2a04d544a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:45.983981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532d3984-94cc-4756-8758-5e2a04d544a7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:45.984265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532d3984-94cc-4756-8758-5e2a04d544a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:45.984534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-532d3984-94cc-4756-8758-5e2a04d544a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:45.995061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532d3984-94cc-4756-8758-5e2a04d544a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:45.996552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-532d3984-94cc-4756-8758-5e2a04d544a7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.000994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-532d3984-94cc-4756-8758-5e2a04d544a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:46.014489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:46.015364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:46.016207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:46.016207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:46.016207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:46.016207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:46.016207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 256/513] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:46.016463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532d3984-94cc-4756-8758-5e2a04d544a7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.039492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b688bf-3f3a-47b1-95e5-9356f5132780 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.075880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7963c6ff-5348-48cb-a473-7549db57cb8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.076679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7963c6ff-5348-48cb-a473-7549db57cb8d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.076989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7963c6ff-5348-48cb-a473-7549db57cb8d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.077269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7963c6ff-5348-48cb-a473-7549db57cb8d None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.084395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7963c6ff-5348-48cb-a473-7549db57cb8d None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.085904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7963c6ff-5348-48cb-a473-7549db57cb8d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.100142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-7963c6ff-5348-48cb-a473-7549db57cb8d None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:46.100373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:46.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 258/514] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:45 2022] GET /identity/v3/users/cinder => generated 84 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:46.102437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7963c6ff-5348-48cb-a473-7549db57cb8d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.120644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-532d3984-94cc-4756-8758-5e2a04d544a7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.154091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f5d36a-2f1f-4c8b-8cff-65c178298d58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.154737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f5d36a-2f1f-4c8b-8cff-65c178298d58 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.155024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f5d36a-2f1f-4c8b-8cff-65c178298d58 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.156159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f5d36a-2f1f-4c8b-8cff-65c178298d58 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.202832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7963c6ff-5348-48cb-a473-7549db57cb8d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.223170 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f5d36a-2f1f-4c8b-8cff-65c178298d58 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.224660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f5d36a-2f1f-4c8b-8cff-65c178298d58 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.224831 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:46.243920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dadd189-ca5c-4b36-b6e7-85ece32bde81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.244647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadd189-ca5c-4b36-b6e7-85ece32bde81 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.244952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadd189-ca5c-4b36-b6e7-85ece32bde81 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.245336 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dadd189-ca5c-4b36-b6e7-85ece32bde81 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.251872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dadd189-ca5c-4b36-b6e7-85ece32bde81 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.253362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dadd189-ca5c-4b36-b6e7-85ece32bde81 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-6dadd189-ca5c-4b36-b6e7-85ece32bde81 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:46.267967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:46.268877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:46.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:46.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:46.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:46.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:46.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:46.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:46.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:46.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:46.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 259/515] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:46.270492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dadd189-ca5c-4b36-b6e7-85ece32bde81 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.308520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:46.326264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 257/516] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:46.327093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f5d36a-2f1f-4c8b-8cff-65c178298d58 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.378865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dadd189-ca5c-4b36-b6e7-85ece32bde81 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.412325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fb2bb1-fd27-4a33-9d7d-2937bbbc4c0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.413005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fb2bb1-fd27-4a33-9d7d-2937bbbc4c0a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.413289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fb2bb1-fd27-4a33-9d7d-2937bbbc4c0a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.413571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fb2bb1-fd27-4a33-9d7d-2937bbbc4c0a None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.414279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fb2bb1-fd27-4a33-9d7d-2937bbbc4c0a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:46.415115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fb2bb1-fd27-4a33-9d7d-2937bbbc4c0a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.416491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fb2bb1-fd27-4a33-9d7d-2937bbbc4c0a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.425364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f5d36a-2f1f-4c8b-8cff-65c178298d58 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.428041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 260/517] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:46.428682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fb2bb1-fd27-4a33-9d7d-2937bbbc4c0a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.461881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be617c38-0a46-4ced-879c-bcc0ba7546e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.462526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be617c38-0a46-4ced-879c-bcc0ba7546e9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.462805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be617c38-0a46-4ced-879c-bcc0ba7546e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.463077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be617c38-0a46-4ced-879c-bcc0ba7546e9 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.463786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be617c38-0a46-4ced-879c-bcc0ba7546e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:46.464085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be617c38-0a46-4ced-879c-bcc0ba7546e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.465481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be617c38-0a46-4ced-879c-bcc0ba7546e9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.469676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:46.476595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 258/518] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:46.478001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be617c38-0a46-4ced-879c-bcc0ba7546e9 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.519821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:46.525103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fb2bb1-fd27-4a33-9d7d-2937bbbc4c0a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.560456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecdf6de-c0a5-46ee-8620-4f9328392375 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.561153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecdf6de-c0a5-46ee-8620-4f9328392375 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.561436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecdf6de-c0a5-46ee-8620-4f9328392375 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.561714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ecdf6de-c0a5-46ee-8620-4f9328392375 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.562543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecdf6de-c0a5-46ee-8620-4f9328392375 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:46.562849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecdf6de-c0a5-46ee-8620-4f9328392375 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.564292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ecdf6de-c0a5-46ee-8620-4f9328392375 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.575831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be617c38-0a46-4ced-879c-bcc0ba7546e9 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.581035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 261/519] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:46.581841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecdf6de-c0a5-46ee-8620-4f9328392375 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.612631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222afea8-014f-47ac-9d5f-2bc53309a32c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.613319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222afea8-014f-47ac-9d5f-2bc53309a32c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.613601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222afea8-014f-47ac-9d5f-2bc53309a32c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.613875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222afea8-014f-47ac-9d5f-2bc53309a32c None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.614651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222afea8-014f-47ac-9d5f-2bc53309a32c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:46.614957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222afea8-014f-47ac-9d5f-2bc53309a32c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.616393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222afea8-014f-47ac-9d5f-2bc53309a32c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.634065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 259/520] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:46.634873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222afea8-014f-47ac-9d5f-2bc53309a32c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.679336 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:46.683337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ecdf6de-c0a5-46ee-8620-4f9328392375 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.716972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becb4ef5-d6c9-42f4-865f-fa21a93f2347 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.717977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becb4ef5-d6c9-42f4-865f-fa21a93f2347 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.718253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becb4ef5-d6c9-42f4-865f-fa21a93f2347 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.718529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becb4ef5-d6c9-42f4-865f-fa21a93f2347 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.728856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becb4ef5-d6c9-42f4-865f-fa21a93f2347 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.730309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becb4ef5-d6c9-42f4-865f-fa21a93f2347 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.740534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222afea8-014f-47ac-9d5f-2bc53309a32c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.742568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-becb4ef5-d6c9-42f4-865f-fa21a93f2347 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:46.748570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:46.759077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:46.760037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:46.760037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:46.760037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:46.760037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:46.760037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 262/521] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:46.760037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becb4ef5-d6c9-42f4-865f-fa21a93f2347 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.776913 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393804d3-68bb-4578-b578-691ac00c0a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.778119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393804d3-68bb-4578-b578-691ac00c0a1e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.778715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393804d3-68bb-4578-b578-691ac00c0a1e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.778994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393804d3-68bb-4578-b578-691ac00c0a1e None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.785953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393804d3-68bb-4578-b578-691ac00c0a1e None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.787388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-393804d3-68bb-4578-b578-691ac00c0a1e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-393804d3-68bb-4578-b578-691ac00c0a1e None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:46.801760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:46.802716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:46.803624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 260/522] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/users/neutron => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:46.804202 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393804d3-68bb-4578-b578-691ac00c0a1e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.851816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becb4ef5-d6c9-42f4-865f-fa21a93f2347 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.885146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a15c031-5294-48bb-8edf-39325ca7b39d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.885809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a15c031-5294-48bb-8edf-39325ca7b39d None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.886094 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a15c031-5294-48bb-8edf-39325ca7b39d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.886373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a15c031-5294-48bb-8edf-39325ca7b39d None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.887068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a15c031-5294-48bb-8edf-39325ca7b39d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:46.888510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a15c031-5294-48bb-8edf-39325ca7b39d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:46.900959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393804d3-68bb-4578-b578-691ac00c0a1e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:46.951556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a88442-3a78-42d6-8d46-99342f0c3fd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:46.952194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a88442-3a78-42d6-8d46-99342f0c3fd1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:46.952480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a88442-3a78-42d6-8d46-99342f0c3fd1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:46.952759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a88442-3a78-42d6-8d46-99342f0c3fd1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:46.971143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 263/523] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:45:46 2022] POST /identity/v3/users => generated 247 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:45:46.971949 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a15c031-5294-48bb-8edf-39325ca7b39d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:46.983848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:46.995654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:47.036417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a88442-3a78-42d6-8d46-99342f0c3fd1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:47.037848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a88442-3a78-42d6-8d46-99342f0c3fd1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:47.084045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a15c031-5294-48bb-8edf-39325ca7b39d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:47.124929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff84545-8103-4581-8c21-0f83d74a3b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:47.125597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff84545-8103-4581-8c21-0f83d74a3b4b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:47.125889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff84545-8103-4581-8c21-0f83d74a3b4b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:47.126163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff84545-8103-4581-8c21-0f83d74a3b4b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:47.127005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff84545-8103-4581-8c21-0f83d74a3b4b None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=e9a0496199134369a0ef740c099eed59) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:47.127318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff84545-8103-4581-8c21-0f83d74a3b4b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:47.128799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff84545-8103-4581-8c21-0f83d74a3b4b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:47.135834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 264/524] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/role_assignments?user.id=e9a0496199134369a0ef740c099eed59&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:47.136641 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff84545-8103-4581-8c21-0f83d74a3b4b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:47.151792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 261/525] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:47.153326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a88442-3a78-42d6-8d46-99342f0c3fd1 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:47.240680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:47.241226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:47.287984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff84545-8103-4581-8c21-0f83d74a3b4b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:47.295488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a88442-3a78-42d6-8d46-99342f0c3fd1 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:47.345919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800fd034-1700-408d-8be3-ea760debf5c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:47.346615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800fd034-1700-408d-8be3-ea760debf5c7 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:47.346917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800fd034-1700-408d-8be3-ea760debf5c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:47.347168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800fd034-1700-408d-8be3-ea760debf5c7 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:47.347911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800fd034-1700-408d-8be3-ea760debf5c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:47.348224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800fd034-1700-408d-8be3-ea760debf5c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:47.349662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800fd034-1700-408d-8be3-ea760debf5c7 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:47.350396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b131a65-a52f-4f8e-99a5-c7c02ad61ad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:47.351693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b131a65-a52f-4f8e-99a5-c7c02ad61ad8 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:47.353393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b131a65-a52f-4f8e-99a5-c7c02ad61ad8 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:47.353878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b131a65-a52f-4f8e-99a5-c7c02ad61ad8 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:47.354915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b131a65-a52f-4f8e-99a5-c7c02ad61ad8 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:47.355415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b131a65-a52f-4f8e-99a5-c7c02ad61ad8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:47.357215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b131a65-a52f-4f8e-99a5-c7c02ad61ad8 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:47.363797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 265/526] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:47.369104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800fd034-1700-408d-8be3-ea760debf5c7 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:47.377166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 262/527] 10.222.0.25 () {56 vars in 1139 bytes} [Fri Jun 10 12:45:47 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:47.378349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b131a65-a52f-4f8e-99a5-c7c02ad61ad8 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:47.486679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b131a65-a52f-4f8e-99a5-c7c02ad61ad8 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:47.496540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800fd034-1700-408d-8be3-ea760debf5c7 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:47.510055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:47.526890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66434a67-3a32-4ff7-a5a5-5b4731d88372 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:47.528896 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66434a67-3a32-4ff7-a5a5-5b4731d88372 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:47.529415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66434a67-3a32-4ff7-a5a5-5b4731d88372 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:47.529860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66434a67-3a32-4ff7-a5a5-5b4731d88372 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:47.534216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:47.539212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f65c1e-e4af-4e18-b508-52d33d030ead None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:47.539642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66434a67-3a32-4ff7-a5a5-5b4731d88372 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:47.540119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f65c1e-e4af-4e18-b508-52d33d030ead None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:47.540476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f65c1e-e4af-4e18-b508-52d33d030ead None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:47.540851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f65c1e-e4af-4e18-b508-52d33d030ead None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:47.542203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66434a67-3a32-4ff7-a5a5-5b4731d88372 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:47.551874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f65c1e-e4af-4e18-b508-52d33d030ead None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:47.553483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f65c1e-e4af-4e18-b508-52d33d030ead None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-66434a67-3a32-4ff7-a5a5-5b4731d88372 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:47.559542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:47.560547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:47.561503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:47.561503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:47.561503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:47.561503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:47.561503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:47.561503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:47.561503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:47.561503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:47.563755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 263/528] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:47.564821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66434a67-3a32-4ff7-a5a5-5b4731d88372 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-34f65c1e-e4af-4e18-b508-52d33d030ead None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:47.580193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:47.581378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:47.582209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:47.582209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:47.582209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:47.582209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:47.582209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 266/529] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:47.582475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f65c1e-e4af-4e18-b508-52d33d030ead None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:47.679823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66434a67-3a32-4ff7-a5a5-5b4731d88372 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:47.694030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f65c1e-e4af-4e18-b508-52d33d030ead None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:47.720422 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbeca62-f7f5-4cb5-8871-aacf9487837b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:47.721426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbeca62-f7f5-4cb5-8871-aacf9487837b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:47.721917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbeca62-f7f5-4cb5-8871-aacf9487837b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:47.722473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbeca62-f7f5-4cb5-8871-aacf9487837b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:47.734327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dca994-b54c-400d-9cc8-57f29b206846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:47.735255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dca994-b54c-400d-9cc8-57f29b206846 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:47.735368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dca994-b54c-400d-9cc8-57f29b206846 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:47.735639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8dca994-b54c-400d-9cc8-57f29b206846 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:47.736429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dca994-b54c-400d-9cc8-57f29b206846 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:47.736740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dca994-b54c-400d-9cc8-57f29b206846 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:47.738438 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8dca994-b54c-400d-9cc8-57f29b206846 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:47.761093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 267/530] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:47.761093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dca994-b54c-400d-9cc8-57f29b206846 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:47.782725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:47.804785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbeca62-f7f5-4cb5-8871-aacf9487837b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:47.806768 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbeca62-f7f5-4cb5-8871-aacf9487837b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:47.813337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:47.867268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8dca994-b54c-400d-9cc8-57f29b206846 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:47.904602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4149a69d-b72c-45f6-9117-ff20145e1500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:47.905299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4149a69d-b72c-45f6-9117-ff20145e1500 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:47.905584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4149a69d-b72c-45f6-9117-ff20145e1500 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:47.905864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4149a69d-b72c-45f6-9117-ff20145e1500 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:47.906750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4149a69d-b72c-45f6-9117-ff20145e1500 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:47.907065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4149a69d-b72c-45f6-9117-ff20145e1500 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:47.908759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4149a69d-b72c-45f6-9117-ff20145e1500 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:47.928354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 268/531] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:47.928919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 264/532] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:47.929187 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4149a69d-b72c-45f6-9117-ff20145e1500 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:47.930484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbeca62-f7f5-4cb5-8871-aacf9487837b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:48.031902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbeca62-f7f5-4cb5-8871-aacf9487837b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:48.059934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4149a69d-b72c-45f6-9117-ff20145e1500 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:48.061090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:48.062959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:48.069503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3199b8b6-efcb-4ee2-8d75-2717f53e18c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:48.070552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199b8b6-efcb-4ee2-8d75-2717f53e18c7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:48.071695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199b8b6-efcb-4ee2-8d75-2717f53e18c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:48.072167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3199b8b6-efcb-4ee2-8d75-2717f53e18c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:48.082592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3199b8b6-efcb-4ee2-8d75-2717f53e18c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:48.084430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3199b8b6-efcb-4ee2-8d75-2717f53e18c7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:48.098710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8ad7a1-e96d-42f4-83bf-74e62a7d2225 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:48.099372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8ad7a1-e96d-42f4-83bf-74e62a7d2225 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:48.099658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8ad7a1-e96d-42f4-83bf-74e62a7d2225 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:48.099937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de8ad7a1-e96d-42f4-83bf-74e62a7d2225 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:48.100875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8ad7a1-e96d-42f4-83bf-74e62a7d2225 None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=4a83b4eb626343f3aa5b1c9f3fc828fd) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:48.111530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8ad7a1-e96d-42f4-83bf-74e62a7d2225 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:48.111530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de8ad7a1-e96d-42f4-83bf-74e62a7d2225 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-3199b8b6-efcb-4ee2-8d75-2717f53e18c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:48.111710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:48.112486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:48.113805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:48.113805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:48.113805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:48.113805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:48.114528 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 265/534] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:48.114759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 269/534] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:47 2022] GET /identity/v3/role_assignments?user.id=4a83b4eb626343f3aa5b1c9f3fc828fd&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:48.115760 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8ad7a1-e96d-42f4-83bf-74e62a7d2225 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:48.116021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3199b8b6-efcb-4ee2-8d75-2717f53e18c7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:48.228472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3199b8b6-efcb-4ee2-8d75-2717f53e18c7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:48.262019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:48.269133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de964735-a9bf-44ea-be67-b026279f33e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:48.270352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de964735-a9bf-44ea-be67-b026279f33e0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:48.270843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de964735-a9bf-44ea-be67-b026279f33e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:48.271323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de964735-a9bf-44ea-be67-b026279f33e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:48.280448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de964735-a9bf-44ea-be67-b026279f33e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:48.282355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de964735-a9bf-44ea-be67-b026279f33e0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-de964735-a9bf-44ea-be67-b026279f33e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:48.299061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:48.299982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:48.301687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:48.301687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:48.301687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:48.301687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:48.301687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:48.301687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:48.301687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:48.301687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:48.305010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 266/535] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:48.318063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de964735-a9bf-44ea-be67-b026279f33e0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:48.326527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de8ad7a1-e96d-42f4-83bf-74e62a7d2225 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:48.408417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04c3c98-c910-4ff8-bb50-508bc9ab9a33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:48.409110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04c3c98-c910-4ff8-bb50-508bc9ab9a33 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:48.409393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04c3c98-c910-4ff8-bb50-508bc9ab9a33 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:48.409667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04c3c98-c910-4ff8-bb50-508bc9ab9a33 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:48.410385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04c3c98-c910-4ff8-bb50-508bc9ab9a33 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:48.410690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04c3c98-c910-4ff8-bb50-508bc9ab9a33 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:48.412069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d04c3c98-c910-4ff8-bb50-508bc9ab9a33 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:48.428805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 270/536] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 310 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:48.431288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04c3c98-c910-4ff8-bb50-508bc9ab9a33 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:48.474160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:48.506963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de964735-a9bf-44ea-be67-b026279f33e0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:48.570011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d04c3c98-c910-4ff8-bb50-508bc9ab9a33 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:48.574814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc70414c-1daa-45bd-b398-25bc308c727f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:48.575481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc70414c-1daa-45bd-b398-25bc308c727f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:48.575763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc70414c-1daa-45bd-b398-25bc308c727f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:48.576039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc70414c-1daa-45bd-b398-25bc308c727f None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:48.576850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc70414c-1daa-45bd-b398-25bc308c727f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:48.577159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc70414c-1daa-45bd-b398-25bc308c727f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:48.578602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc70414c-1daa-45bd-b398-25bc308c727f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:48.603042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 267/537] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:48.609335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc70414c-1daa-45bd-b398-25bc308c727f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:48.609962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0a7bae-1641-481b-9c95-f40c0a4829ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:48.610821 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0a7bae-1641-481b-9c95-f40c0a4829ab None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:48.611103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0a7bae-1641-481b-9c95-f40c0a4829ab None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:48.611379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0a7bae-1641-481b-9c95-f40c0a4829ab None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:48.619064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0a7bae-1641-481b-9c95-f40c0a4829ab None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:48.620532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0a7bae-1641-481b-9c95-f40c0a4829ab None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-ba0a7bae-1641-481b-9c95-f40c0a4829ab None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:48.635184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:48.636032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:48.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:48.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:48.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:48.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:48.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:48.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:48.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:48.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:48.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 271/538] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:48.643767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0a7bae-1641-481b-9c95-f40c0a4829ab None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:48.712200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:48.787254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc70414c-1daa-45bd-b398-25bc308c727f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:48.793405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:48.806104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0a7bae-1641-481b-9c95-f40c0a4829ab None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:48.832754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3249dd3-00c6-4555-ba64-16abe4ff93e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:48.833912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3249dd3-00c6-4555-ba64-16abe4ff93e7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:48.834397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3249dd3-00c6-4555-ba64-16abe4ff93e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:48.834893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3249dd3-00c6-4555-ba64-16abe4ff93e7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:48.835922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3249dd3-00c6-4555-ba64-16abe4ff93e7 None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=184bf14f33464130ab62ed1853bf4f8d) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:48.836493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3249dd3-00c6-4555-ba64-16abe4ff93e7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:48.840275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3249dd3-00c6-4555-ba64-16abe4ff93e7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:48.853347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 268/539] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:48 2022] GET /identity/v3/role_assignments?user.id=184bf14f33464130ab62ed1853bf4f8d&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 259 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:48.860918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b8d10c-363e-432f-a76a-ec0f15d58c41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:48.861971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b8d10c-363e-432f-a76a-ec0f15d58c41 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:48.862456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b8d10c-363e-432f-a76a-ec0f15d58c41 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:48.862992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b8d10c-363e-432f-a76a-ec0f15d58c41 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:48.864017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b8d10c-363e-432f-a76a-ec0f15d58c41 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:48.864584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b8d10c-363e-432f-a76a-ec0f15d58c41 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:48.874772 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b8d10c-363e-432f-a76a-ec0f15d58c41 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:48.889833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 272/540] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:48.896383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3249dd3-00c6-4555-ba64-16abe4ff93e7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:49.028116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3249dd3-00c6-4555-ba64-16abe4ff93e7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:49.046258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:49.064912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92908bfa-e865-45c4-b50e-334a153af8bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:49.065568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92908bfa-e865-45c4-b50e-334a153af8bb None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:49.065849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92908bfa-e865-45c4-b50e-334a153af8bb None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:49.066124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92908bfa-e865-45c4-b50e-334a153af8bb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:49.066954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92908bfa-e865-45c4-b50e-334a153af8bb None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=d6794257c90e40bca2c8efd31b0c791a) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:49.067257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92908bfa-e865-45c4-b50e-334a153af8bb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:49.068812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92908bfa-e865-45c4-b50e-334a153af8bb None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:49.074966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 269/541] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:48 2022] GET /identity/v3/role_assignments?user.id=d6794257c90e40bca2c8efd31b0c791a&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 259 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:49.411002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea3f87-f2f9-4e35-92ca-42d7d139da44 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:49.411529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea3f87-f2f9-4e35-92ca-42d7d139da44 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:49.411529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddea3f87-f2f9-4e35-92ca-42d7d139da44 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:49.413199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 273/542] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:49.419354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef52b1c1-4f01-4cbe-a864-3924869680b5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:49.420180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef52b1c1-4f01-4cbe-a864-3924869680b5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:49.420695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef52b1c1-4f01-4cbe-a864-3924869680b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:49.438353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ef52b1c1-4f01-4cbe-a864-3924869680b5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:49.439908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ef52b1c1-4f01-4cbe-a864-3924869680b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:49.527736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 270/543] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:49.578420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea3f87-f2f9-4e35-92ca-42d7d139da44 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:49.613641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999a712f-b34a-46eb-a0c7-8117317bd9ff None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:49.613923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999a712f-b34a-46eb-a0c7-8117317bd9ff None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:49.614164 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-999a712f-b34a-46eb-a0c7-8117317bd9ff None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:49.615532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 271/544] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:49.628209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e59e0-e615-4d04-af84-3940fa464eec None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:49.628492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e59e0-e615-4d04-af84-3940fa464eec None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:49.628807 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e59e0-e615-4d04-af84-3940fa464eec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:49.665900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-980e59e0-e615-4d04-af84-3940fa464eec None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:49.667113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-980e59e0-e615-4d04-af84-3940fa464eec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:49.722401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddea3f87-f2f9-4e35-92ca-42d7d139da44 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:49.761987 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086a2b5a-458d-41d2-a304-9e1771d87bdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:49.763585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086a2b5a-458d-41d2-a304-9e1771d87bdb None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:49.764011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086a2b5a-458d-41d2-a304-9e1771d87bdb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:49.764480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086a2b5a-458d-41d2-a304-9e1771d87bdb None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:49.776200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086a2b5a-458d-41d2-a304-9e1771d87bdb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-086a2b5a-458d-41d2-a304-9e1771d87bdb None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:49.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:49.786010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:49.786975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:49.790457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 274/545] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:49.795861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086a2b5a-458d-41d2-a304-9e1771d87bdb None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:49.813152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:49.841020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 272/546] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:49.865573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e59e0-e615-4d04-af84-3940fa464eec None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:49.924028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:49.964658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086a2b5a-458d-41d2-a304-9e1771d87bdb None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:50.015904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ab9352-1aa1-406c-98c2-760540a7bc0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:50.016582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ab9352-1aa1-406c-98c2-760540a7bc0a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.016932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ab9352-1aa1-406c-98c2-760540a7bc0a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.017169 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ab9352-1aa1-406c-98c2-760540a7bc0a None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.017931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ab9352-1aa1-406c-98c2-760540a7bc0a None admin] RBAC: Adding query filter params (name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:50.018242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ab9352-1aa1-406c-98c2-760540a7bc0a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:50.019647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ab9352-1aa1-406c-98c2-760540a7bc0a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:50.034305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e59e0-e615-4d04-af84-3940fa464eec None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:50.062638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 275/547] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:50.067922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ab9352-1aa1-406c-98c2-760540a7bc0a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:50.092533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd2af0-19a9-4590-ac0e-db8b2b86a3fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:50.093701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd2af0-19a9-4590-ac0e-db8b2b86a3fe None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.094200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd2af0-19a9-4590-ac0e-db8b2b86a3fe None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.094663 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fd2af0-19a9-4590-ac0e-db8b2b86a3fe None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.122546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fd2af0-19a9-4590-ac0e-db8b2b86a3fe None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-e5fd2af0-19a9-4590-ac0e-db8b2b86a3fe None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:50.130348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:50.131320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:50.132093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 273/548] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:50.139726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd2af0-19a9-4590-ac0e-db8b2b86a3fe None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:50.180638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:50.234291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ab9352-1aa1-406c-98c2-760540a7bc0a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:50.295621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3449a1ac-1d5d-42a1-93b0-f7a65726d4a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:50.296658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3449a1ac-1d5d-42a1-93b0-f7a65726d4a2 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.296964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3449a1ac-1d5d-42a1-93b0-f7a65726d4a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.298035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3449a1ac-1d5d-42a1-93b0-f7a65726d4a2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.311853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:50.315571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fd2af0-19a9-4590-ac0e-db8b2b86a3fe None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:50.324357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3449a1ac-1d5d-42a1-93b0-f7a65726d4a2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:50.325316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3449a1ac-1d5d-42a1-93b0-f7a65726d4a2 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-3449a1ac-1d5d-42a1-93b0-f7a65726d4a2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:50.354911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:50.355807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:50.356682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:50.356682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:50.356682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:50.356682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:50.357467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 276/549] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:50.362517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3449a1ac-1d5d-42a1-93b0-f7a65726d4a2 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:50.381158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577cb412-1423-4b7c-9f79-12f4803a9672 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:50.381773 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577cb412-1423-4b7c-9f79-12f4803a9672 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.382083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577cb412-1423-4b7c-9f79-12f4803a9672 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.382363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577cb412-1423-4b7c-9f79-12f4803a9672 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.383715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577cb412-1423-4b7c-9f79-12f4803a9672 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:50.384042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577cb412-1423-4b7c-9f79-12f4803a9672 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:50.385525 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577cb412-1423-4b7c-9f79-12f4803a9672 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:50.397458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 274/550] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:50.404321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577cb412-1423-4b7c-9f79-12f4803a9672 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:50.554611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3449a1ac-1d5d-42a1-93b0-f7a65726d4a2 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:50.600159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:50.627270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b780460-e026-405c-9814-af587745d06b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:50.627956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b780460-e026-405c-9814-af587745d06b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.628233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b780460-e026-405c-9814-af587745d06b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.630258 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b780460-e026-405c-9814-af587745d06b None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.631049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b780460-e026-405c-9814-af587745d06b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:50.631415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b780460-e026-405c-9814-af587745d06b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:50.632792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b780460-e026-405c-9814-af587745d06b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:50.638913 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577cb412-1423-4b7c-9f79-12f4803a9672 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:50.659046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 277/551] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:50.664982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b780460-e026-405c-9814-af587745d06b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:50.695527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4762144b-5929-46db-9907-72fada765081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:50.696456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4762144b-5929-46db-9907-72fada765081 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.696730 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4762144b-5929-46db-9907-72fada765081 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.697047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4762144b-5929-46db-9907-72fada765081 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.708396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4762144b-5929-46db-9907-72fada765081 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:50.709896 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4762144b-5929-46db-9907-72fada765081 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-4762144b-5929-46db-9907-72fada765081 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:50.728350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:50.729264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:50.730294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:50.730294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:50.730294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:50.730294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:50.730294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 275/552] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:50.731141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bc3b47-7947-4055-b800-d38abed1fce9 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.731426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bc3b47-7947-4055-b800-d38abed1fce9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.731707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6bc3b47-7947-4055-b800-d38abed1fce9 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.737027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 276/553] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:50.737027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bc3b47-7947-4055-b800-d38abed1fce9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:50.763412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:50.818404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b780460-e026-405c-9814-af587745d06b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:50.868772 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34a300c-2e3a-4b13-9ceb-07439ffd76a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:50.869635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34a300c-2e3a-4b13-9ceb-07439ffd76a3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.869924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34a300c-2e3a-4b13-9ceb-07439ffd76a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.870202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34a300c-2e3a-4b13-9ceb-07439ffd76a3 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.879296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34a300c-2e3a-4b13-9ceb-07439ffd76a3 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:50.880790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34a300c-2e3a-4b13-9ceb-07439ffd76a3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:50.894897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6bc3b47-7947-4055-b800-d38abed1fce9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-c34a300c-2e3a-4b13-9ceb-07439ffd76a3 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:50.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:50.899631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:50.900537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 278/554] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:45:50 2022] GET /identity/v3/users/swift => generated 83 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:50.901610 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9f993-6dc6-42ec-ba0e-6e5fdb4881f7 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.901610 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9f993-6dc6-42ec-ba0e-6e5fdb4881f7 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.901740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc9f993-6dc6-42ec-ba0e-6e5fdb4881f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.918414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-4cc9f993-6dc6-42ec-ba0e-6e5fdb4881f7 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:50.919609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4cc9f993-6dc6-42ec-ba0e-6e5fdb4881f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:50.937676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181c4eb2-6d1d-48a3-9e13-849536cedd3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:50.938711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181c4eb2-6d1d-48a3-9e13-849536cedd3c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:50.939219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181c4eb2-6d1d-48a3-9e13-849536cedd3c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:50.939717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181c4eb2-6d1d-48a3-9e13-849536cedd3c None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:50.940799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181c4eb2-6d1d-48a3-9e13-849536cedd3c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:50.941365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181c4eb2-6d1d-48a3-9e13-849536cedd3c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:50.943002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181c4eb2-6d1d-48a3-9e13-849536cedd3c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:50.956692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 277/555] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:50.957237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:50.958439 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181c4eb2-6d1d-48a3-9e13-849536cedd3c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:50.991070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 279/556] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:50.991939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9f993-6dc6-42ec-ba0e-6e5fdb4881f7 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:51.040753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:51.066858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181c4eb2-6d1d-48a3-9e13-849536cedd3c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.108495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc9fe6c-1d0d-4f26-9c85-5974d803e3e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.109568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc9fe6c-1d0d-4f26-9c85-5974d803e3e3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.110069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc9fe6c-1d0d-4f26-9c85-5974d803e3e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.110544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc9fe6c-1d0d-4f26-9c85-5974d803e3e3 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.111191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc9f993-6dc6-42ec-ba0e-6e5fdb4881f7 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.111806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc9fe6c-1d0d-4f26-9c85-5974d803e3e3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:51.112332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc9fe6c-1d0d-4f26-9c85-5974d803e3e3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:51.114020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc9fe6c-1d0d-4f26-9c85-5974d803e3e3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:51.133670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 278/557] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Jun 10 12:45:50 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:51.134903 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc9fe6c-1d0d-4f26-9c85-5974d803e3e3 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:51.148429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f57a7db-2af4-4ed3-ac9b-e3564e9da637 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.149166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f57a7db-2af4-4ed3-ac9b-e3564e9da637 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.149460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f57a7db-2af4-4ed3-ac9b-e3564e9da637 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.149805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f57a7db-2af4-4ed3-ac9b-e3564e9da637 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.157634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f57a7db-2af4-4ed3-ac9b-e3564e9da637 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:51.159085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f57a7db-2af4-4ed3-ac9b-e3564e9da637 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-8f57a7db-2af4-4ed3-ac9b-e3564e9da637 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:51.174235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:51.175709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 280/558] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Jun 10 12:45:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:51.176638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f57a7db-2af4-4ed3-ac9b-e3564e9da637 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:51.210420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:51.248267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc9fe6c-1d0d-4f26-9c85-5974d803e3e3 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.280397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:51.286327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c21933-0061-4633-a853-6ea715fb7a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.287001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f57a7db-2af4-4ed3-ac9b-e3564e9da637 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.287523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c21933-0061-4633-a853-6ea715fb7a0a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.287955 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c21933-0061-4633-a853-6ea715fb7a0a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.288376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28c21933-0061-4633-a853-6ea715fb7a0a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.299529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28c21933-0061-4633-a853-6ea715fb7a0a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-28c21933-0061-4633-a853-6ea715fb7a0a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:51.308248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:51.309425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:51.310581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:51.313029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 279/559] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:51.313981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c21933-0061-4633-a853-6ea715fb7a0a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:51.339316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20999de-66fc-44df-8f9e-6f90413cd77d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.339717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20999de-66fc-44df-8f9e-6f90413cd77d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.340076 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20999de-66fc-44df-8f9e-6f90413cd77d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.340431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20999de-66fc-44df-8f9e-6f90413cd77d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.429544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28c21933-0061-4633-a853-6ea715fb7a0a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.430649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20999de-66fc-44df-8f9e-6f90413cd77d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:51.434255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20999de-66fc-44df-8f9e-6f90413cd77d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:51.478386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d9e85f-9a5b-4c11-938e-1eb12c842447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.479424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9e85f-9a5b-4c11-938e-1eb12c842447 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.479893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9e85f-9a5b-4c11-938e-1eb12c842447 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.480381 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d9e85f-9a5b-4c11-938e-1eb12c842447 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.481503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d9e85f-9a5b-4c11-938e-1eb12c842447 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:51.482042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d9e85f-9a5b-4c11-938e-1eb12c842447 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:51.483701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8d9e85f-9a5b-4c11-938e-1eb12c842447 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:51.488950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:51.503549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 280/560] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Jun 10 12:45:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:51.504802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d9e85f-9a5b-4c11-938e-1eb12c842447 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:51.558769 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:51.579330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 281/561] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 398 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:51.579330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aec2440-6e94-4fc3-a4f6-e42e16da875c None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.579330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aec2440-6e94-4fc3-a4f6-e42e16da875c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.579330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aec2440-6e94-4fc3-a4f6-e42e16da875c None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.579330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 282/562] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:51.579330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aec2440-6e94-4fc3-a4f6-e42e16da875c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:51.620380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8d9e85f-9a5b-4c11-938e-1eb12c842447 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.655167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864d56e5-364c-44ff-89c6-b493d380ab0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.655838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864d56e5-364c-44ff-89c6-b493d380ab0e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.656090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864d56e5-364c-44ff-89c6-b493d380ab0e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.656366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864d56e5-364c-44ff-89c6-b493d380ab0e None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.657139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864d56e5-364c-44ff-89c6-b493d380ab0e None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:51.657516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864d56e5-364c-44ff-89c6-b493d380ab0e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:51.658919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864d56e5-364c-44ff-89c6-b493d380ab0e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:51.667129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 281/563] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:51.668213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864d56e5-364c-44ff-89c6-b493d380ab0e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:51.687331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aec2440-6e94-4fc3-a4f6-e42e16da875c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.716885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:51.721627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3e7831-e6f5-49ec-8c0b-d0591c4f0953 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.722417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e7831-e6f5-49ec-8c0b-d0591c4f0953 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.722702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e7831-e6f5-49ec-8c0b-d0591c4f0953 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.722986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3e7831-e6f5-49ec-8c0b-d0591c4f0953 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.733536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3e7831-e6f5-49ec-8c0b-d0591c4f0953 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:51.735007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3e7831-e6f5-49ec-8c0b-d0591c4f0953 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-9b3e7831-e6f5-49ec-8c0b-d0591c4f0953 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:51.752698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:51.753801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:51.755020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:51.755020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:51.755020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:51.755020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:51.755020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 283/564] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:51.755355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953ba1d2-8ba3-48ed-995e-11365d45ecc5 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.755355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953ba1d2-8ba3-48ed-995e-11365d45ecc5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.755891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-953ba1d2-8ba3-48ed-995e-11365d45ecc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.768024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864d56e5-364c-44ff-89c6-b493d380ab0e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.771705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-953ba1d2-8ba3-48ed-995e-11365d45ecc5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:51.772905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-953ba1d2-8ba3-48ed-995e-11365d45ecc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:51.781413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:51.800951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609503b1-6559-414a-bd92-3e4e6e997cfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.801737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609503b1-6559-414a-bd92-3e4e6e997cfe None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.802019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609503b1-6559-414a-bd92-3e4e6e997cfe None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.802297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609503b1-6559-414a-bd92-3e4e6e997cfe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.812444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609503b1-6559-414a-bd92-3e4e6e997cfe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:51.813936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609503b1-6559-414a-bd92-3e4e6e997cfe None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-609503b1-6559-414a-bd92-3e4e6e997cfe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:51.831440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:51.832375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:51.833294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:51.833294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:51.833294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:51.833294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:51.833294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 282/565] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:51.834016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6cd0f1-c084-4177-bb2c-a4dd5d3e3541 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.834270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6cd0f1-c084-4177-bb2c-a4dd5d3e3541 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.834548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6cd0f1-c084-4177-bb2c-a4dd5d3e3541 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.835988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 283/566] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:51.837142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6cd0f1-c084-4177-bb2c-a4dd5d3e3541 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:51.846119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 284/567] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:51.847028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953ba1d2-8ba3-48ed-995e-11365d45ecc5 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:51.950904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6cd0f1-c084-4177-bb2c-a4dd5d3e3541 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.952837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-953ba1d2-8ba3-48ed-995e-11365d45ecc5 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:51.968836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:51.984660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545f5234-3a05-4960-a4aa-a857850491be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.985471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545f5234-3a05-4960-a4aa-a857850491be None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.985748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545f5234-3a05-4960-a4aa-a857850491be None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.986017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545f5234-3a05-4960-a4aa-a857850491be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.987959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2c8a0-fee0-415d-9fab-798bb2f000b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:51.988613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2c8a0-fee0-415d-9fab-798bb2f000b1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:51.988922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2c8a0-fee0-415d-9fab-798bb2f000b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:51.989195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f2c8a0-fee0-415d-9fab-798bb2f000b1 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:51.989899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2c8a0-fee0-415d-9fab-798bb2f000b1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:51.990201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2c8a0-fee0-415d-9fab-798bb2f000b1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:51.991582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f2c8a0-fee0-415d-9fab-798bb2f000b1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:51.996901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545f5234-3a05-4960-a4aa-a857850491be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:51.998334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545f5234-3a05-4960-a4aa-a857850491be None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.003174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 285/568] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:52.003961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2c8a0-fee0-415d-9fab-798bb2f000b1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-545f5234-3a05-4960-a4aa-a857850491be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:52.017272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:52.018231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:52.019142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:52.019142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:52.019142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:52.019142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:52.019142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 284/569] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:52.019637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6c5c4-4dad-43e2-905e-1fdb3b66c00e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.019915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6c5c4-4dad-43e2-905e-1fdb3b66c00e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.020185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79b6c5c4-4dad-43e2-905e-1fdb3b66c00e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.036937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-79b6c5c4-4dad-43e2-905e-1fdb3b66c00e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:52.038122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-79b6c5c4-4dad-43e2-905e-1fdb3b66c00e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:52.048610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:52.104806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 285/570] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:52.107125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f2c8a0-fee0-415d-9fab-798bb2f000b1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.118369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b6c5c4-4dad-43e2-905e-1fdb3b66c00e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.141426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd576905-3c54-4c2e-a3e1-98f048c58c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.142087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd576905-3c54-4c2e-a3e1-98f048c58c24 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.142373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd576905-3c54-4c2e-a3e1-98f048c58c24 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.142661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd576905-3c54-4c2e-a3e1-98f048c58c24 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.143374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd576905-3c54-4c2e-a3e1-98f048c58c24 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:52.143685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd576905-3c54-4c2e-a3e1-98f048c58c24 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:52.145106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd576905-3c54-4c2e-a3e1-98f048c58c24 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.166896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 286/571] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:52.168388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd576905-3c54-4c2e-a3e1-98f048c58c24 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.206699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:52.231124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79b6c5c4-4dad-43e2-905e-1fdb3b66c00e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.264626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c68b51f-c022-4f1f-a01f-280259474f70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.265451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c68b51f-c022-4f1f-a01f-280259474f70 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.265740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c68b51f-c022-4f1f-a01f-280259474f70 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.266016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c68b51f-c022-4f1f-a01f-280259474f70 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.271105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd576905-3c54-4c2e-a3e1-98f048c58c24 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.276501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c68b51f-c022-4f1f-a01f-280259474f70 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-8c68b51f-c022-4f1f-a01f-280259474f70 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:52.284427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:52.285394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:52.286392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 286/572] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:52.287217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c68b51f-c022-4f1f-a01f-280259474f70 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.296291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:52.310360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1a100d-eb90-4bb5-a83d-b18ab5f0ef6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.311151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a100d-eb90-4bb5-a83d-b18ab5f0ef6d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.311433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a100d-eb90-4bb5-a83d-b18ab5f0ef6d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.311709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1a100d-eb90-4bb5-a83d-b18ab5f0ef6d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.318449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1a100d-eb90-4bb5-a83d-b18ab5f0ef6d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:52.320343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa1a100d-eb90-4bb5-a83d-b18ab5f0ef6d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-fa1a100d-eb90-4bb5-a83d-b18ab5f0ef6d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:52.334395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:52.335305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 287/573] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:52.337958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1a100d-eb90-4bb5-a83d-b18ab5f0ef6d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.388700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c68b51f-c022-4f1f-a01f-280259474f70 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.421569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1dd573-fdeb-4ec8-93de-8032ea5dd4d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.422227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1dd573-fdeb-4ec8-93de-8032ea5dd4d1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.422515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1dd573-fdeb-4ec8-93de-8032ea5dd4d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.422790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1dd573-fdeb-4ec8-93de-8032ea5dd4d1 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.423505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1dd573-fdeb-4ec8-93de-8032ea5dd4d1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:52.423813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1dd573-fdeb-4ec8-93de-8032ea5dd4d1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:52.425248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1dd573-fdeb-4ec8-93de-8032ea5dd4d1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.428021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:52.434955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1a100d-eb90-4bb5-a83d-b18ab5f0ef6d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.437512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 287/574] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:52.438305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1dd573-fdeb-4ec8-93de-8032ea5dd4d1 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.475810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267899ae-1913-4985-bae6-37a96987d601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.476531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267899ae-1913-4985-bae6-37a96987d601 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.477379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267899ae-1913-4985-bae6-37a96987d601 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.477781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267899ae-1913-4985-bae6-37a96987d601 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.488998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267899ae-1913-4985-bae6-37a96987d601 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-267899ae-1913-4985-bae6-37a96987d601 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:52.498183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:52.499318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 288/575] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:52.500148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267899ae-1913-4985-bae6-37a96987d601 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.517863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:52.552305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1dd573-fdeb-4ec8-93de-8032ea5dd4d1 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.587159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e4dc5e-bcec-40ec-84e6-052c52b4251f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.588119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e4dc5e-bcec-40ec-84e6-052c52b4251f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.588482 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e4dc5e-bcec-40ec-84e6-052c52b4251f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.588787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e4dc5e-bcec-40ec-84e6-052c52b4251f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.597430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e4dc5e-bcec-40ec-84e6-052c52b4251f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:52.600101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e4dc5e-bcec-40ec-84e6-052c52b4251f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.601186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267899ae-1913-4985-bae6-37a96987d601 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-12e4dc5e-bcec-40ec-84e6-052c52b4251f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:52.615114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:52.616157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:52.617230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:52.617230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:52.617230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:52.617230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:52.617230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:52.617230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:52.617230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:52.617230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:52.617230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 288/576] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:52.618384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e4dc5e-bcec-40ec-84e6-052c52b4251f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.637695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bf723d-d29e-4e1a-8456-4c9a9455373e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.638363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bf723d-d29e-4e1a-8456-4c9a9455373e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.638649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bf723d-d29e-4e1a-8456-4c9a9455373e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.638924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49bf723d-d29e-4e1a-8456-4c9a9455373e None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.639663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bf723d-d29e-4e1a-8456-4c9a9455373e None admin] RBAC: Adding query filter params (name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:52.639974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bf723d-d29e-4e1a-8456-4c9a9455373e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:52.641389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49bf723d-d29e-4e1a-8456-4c9a9455373e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.649368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 289/577] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:52.649992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bf723d-d29e-4e1a-8456-4c9a9455373e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.668633 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:52.719259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e4dc5e-bcec-40ec-84e6-052c52b4251f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.746464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:52.751509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49bf723d-d29e-4e1a-8456-4c9a9455373e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.755513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb035f5-2907-443d-bbf2-f26926e0c90a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.763541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb035f5-2907-443d-bbf2-f26926e0c90a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.763541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb035f5-2907-443d-bbf2-f26926e0c90a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.763541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb035f5-2907-443d-bbf2-f26926e0c90a None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.763541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb035f5-2907-443d-bbf2-f26926e0c90a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:52.763541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb035f5-2907-443d-bbf2-f26926e0c90a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:52.763541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb035f5-2907-443d-bbf2-f26926e0c90a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.770471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 289/578] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:52.771248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb035f5-2907-443d-bbf2-f26926e0c90a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.789869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0748e8-714a-428f-b3af-7535adff1023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.790691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0748e8-714a-428f-b3af-7535adff1023 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.790977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0748e8-714a-428f-b3af-7535adff1023 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.791256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0748e8-714a-428f-b3af-7535adff1023 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.799067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0748e8-714a-428f-b3af-7535adff1023 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:52.800510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0748e8-714a-428f-b3af-7535adff1023 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-1b0748e8-714a-428f-b3af-7535adff1023 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:52.814957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:52.815907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:52.816893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 290/579] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/users/glance => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:52.817939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0748e8-714a-428f-b3af-7535adff1023 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.873241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb035f5-2907-443d-bbf2-f26926e0c90a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.891162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:52.906471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501a1e86-fe20-4d98-97c8-cce229e7527b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.907134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501a1e86-fe20-4d98-97c8-cce229e7527b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.907421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501a1e86-fe20-4d98-97c8-cce229e7527b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.907702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501a1e86-fe20-4d98-97c8-cce229e7527b None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.908490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501a1e86-fe20-4d98-97c8-cce229e7527b None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:52.908823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501a1e86-fe20-4d98-97c8-cce229e7527b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:52.910198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501a1e86-fe20-4d98-97c8-cce229e7527b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.913748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0748e8-714a-428f-b3af-7535adff1023 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:52.918113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 290/580] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:52.918907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501a1e86-fe20-4d98-97c8-cce229e7527b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.947205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dbd194-4872-4d21-a626-b8c786cca34a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:52.947854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbd194-4872-4d21-a626-b8c786cca34a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:52.948131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbd194-4872-4d21-a626-b8c786cca34a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:52.948410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78dbd194-4872-4d21-a626-b8c786cca34a None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:52.949208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dbd194-4872-4d21-a626-b8c786cca34a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:52.949519 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dbd194-4872-4d21-a626-b8c786cca34a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:52.950955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78dbd194-4872-4d21-a626-b8c786cca34a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:52.963514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 291/581] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:52.964345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dbd194-4872-4d21-a626-b8c786cca34a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:52.970162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:53.069349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501a1e86-fe20-4d98-97c8-cce229e7527b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.102635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78dbd194-4872-4d21-a626-b8c786cca34a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.103826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2687bce0-4615-4ba9-9a88-cb173b4bd393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.104609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2687bce0-4615-4ba9-9a88-cb173b4bd393 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.104932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2687bce0-4615-4ba9-9a88-cb173b4bd393 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.105210 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2687bce0-4615-4ba9-9a88-cb173b4bd393 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.115437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2687bce0-4615-4ba9-9a88-cb173b4bd393 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.116917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2687bce0-4615-4ba9-9a88-cb173b4bd393 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-2687bce0-4615-4ba9-9a88-cb173b4bd393 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:53.134459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:53.135400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:53.141239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:53.141239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:53.141239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:53.141239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:53.141239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 291/582] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:53.141239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2687bce0-4615-4ba9-9a88-cb173b4bd393 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:53.141578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4c6131-6eff-4a0d-8493-574d0b901201 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.141578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4c6131-6eff-4a0d-8493-574d0b901201 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.141578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4c6131-6eff-4a0d-8493-574d0b901201 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.141578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4c6131-6eff-4a0d-8493-574d0b901201 None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/e9a0496199134369a0ef740c099eed59/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.142467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:53.153670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4c6131-6eff-4a0d-8493-574d0b901201 None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=e9a0496199134369a0ef740c099eed59, role_id=c7c592df9b6b4743a1ba6ca3b2a7fdbc)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.155118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4c6131-6eff-4a0d-8493-574d0b901201 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:53.228893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 292/583] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:45:52 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/e9a0496199134369a0ef740c099eed59/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:53.229697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4c6131-6eff-4a0d-8493-574d0b901201 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:53.230401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:53.233897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2687bce0-4615-4ba9-9a88-cb173b4bd393 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.271705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f73667-9001-4499-bac5-d06ab86c6168 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.272389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f73667-9001-4499-bac5-d06ab86c6168 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.272673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f73667-9001-4499-bac5-d06ab86c6168 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.272984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f73667-9001-4499-bac5-d06ab86c6168 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.274487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f73667-9001-4499-bac5-d06ab86c6168 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:53.274811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f73667-9001-4499-bac5-d06ab86c6168 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.276298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f73667-9001-4499-bac5-d06ab86c6168 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:53.301706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 292/584] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Jun 10 12:45:53 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:53.301706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f73667-9001-4499-bac5-d06ab86c6168 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:53.365965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4c6131-6eff-4a0d-8493-574d0b901201 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.407811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828e94db-6210-428d-9e1c-0144d230cce1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.408606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828e94db-6210-428d-9e1c-0144d230cce1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.408977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828e94db-6210-428d-9e1c-0144d230cce1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.409254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828e94db-6210-428d-9e1c-0144d230cce1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.418418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f73667-9001-4499-bac5-d06ab86c6168 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.419958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828e94db-6210-428d-9e1c-0144d230cce1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.421445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-828e94db-6210-428d-9e1c-0144d230cce1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-828e94db-6210-428d-9e1c-0144d230cce1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:53.439064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:53.440000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:53.440921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:53.440921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:53.440921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:53.440921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:53.440921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 293/585] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:53.442096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828e94db-6210-428d-9e1c-0144d230cce1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:53.457645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f5433a-9b5b-4f98-b7fa-a54c3d0a8ca3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.458401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f5433a-9b5b-4f98-b7fa-a54c3d0a8ca3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.458688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f5433a-9b5b-4f98-b7fa-a54c3d0a8ca3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.459022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f5433a-9b5b-4f98-b7fa-a54c3d0a8ca3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.459793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f5433a-9b5b-4f98-b7fa-a54c3d0a8ca3 None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=bb18cd41c9a14d358aac14ed62ef9e02) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:53.460107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f5433a-9b5b-4f98-b7fa-a54c3d0a8ca3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.463798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f5433a-9b5b-4f98-b7fa-a54c3d0a8ca3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:53.480791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 293/586] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:53 2022] GET /identity/v3/role_assignments?user.id=bb18cd41c9a14d358aac14ed62ef9e02&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 259 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:53.480791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f5433a-9b5b-4f98-b7fa-a54c3d0a8ca3 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:53.494027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:53.512470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:53.574625 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828e94db-6210-428d-9e1c-0144d230cce1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.601359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f5433a-9b5b-4f98-b7fa-a54c3d0a8ca3 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.613766 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eaa3d84-7405-41b8-95ca-7ac36c8ecc8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.614447 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaa3d84-7405-41b8-95ca-7ac36c8ecc8b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.614724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaa3d84-7405-41b8-95ca-7ac36c8ecc8b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.615005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eaa3d84-7405-41b8-95ca-7ac36c8ecc8b None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.615745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eaa3d84-7405-41b8-95ca-7ac36c8ecc8b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:53.616057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eaa3d84-7405-41b8-95ca-7ac36c8ecc8b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.618029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eaa3d84-7405-41b8-95ca-7ac36c8ecc8b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:53.630662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 294/587] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:53.631536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eaa3d84-7405-41b8-95ca-7ac36c8ecc8b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:53.645211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2e2c18-3c00-47cc-afc1-80e5ad70e9f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.646055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2e2c18-3c00-47cc-afc1-80e5ad70e9f4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.646391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2e2c18-3c00-47cc-afc1-80e5ad70e9f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.646676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2e2c18-3c00-47cc-afc1-80e5ad70e9f4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.658756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2e2c18-3c00-47cc-afc1-80e5ad70e9f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.660403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2e2c18-3c00-47cc-afc1-80e5ad70e9f4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-5d2e2c18-3c00-47cc-afc1-80e5ad70e9f4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:53.684682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:53.685693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:53.686608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:53.686608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:53.686608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:53.686608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:53.691403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 294/588] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:53.692289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2e2c18-3c00-47cc-afc1-80e5ad70e9f4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:53.735865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:53.739831 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eaa3d84-7405-41b8-95ca-7ac36c8ecc8b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.777174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed54528d-8880-4a74-89bb-41ec0841ed38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.778006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54528d-8880-4a74-89bb-41ec0841ed38 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.778394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54528d-8880-4a74-89bb-41ec0841ed38 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.778797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed54528d-8880-4a74-89bb-41ec0841ed38 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.779738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed54528d-8880-4a74-89bb-41ec0841ed38 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:53.780151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed54528d-8880-4a74-89bb-41ec0841ed38 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.781737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed54528d-8880-4a74-89bb-41ec0841ed38 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:53.790212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:53.795879 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 295/589] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:53.796967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed54528d-8880-4a74-89bb-41ec0841ed38 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:53.826110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2e2c18-3c00-47cc-afc1-80e5ad70e9f4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.868144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e472d9-f4f5-432d-b53d-f4f684c4307b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.868947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e472d9-f4f5-432d-b53d-f4f684c4307b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.869353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e472d9-f4f5-432d-b53d-f4f684c4307b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.869632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e472d9-f4f5-432d-b53d-f4f684c4307b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.879130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e472d9-f4f5-432d-b53d-f4f684c4307b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.880752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e472d9-f4f5-432d-b53d-f4f684c4307b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-e8e472d9-f4f5-432d-b53d-f4f684c4307b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:53.898917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:53.899997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:53.901007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 295/590] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:53 2022] GET /identity/v3/users/cinder => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:53.901595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e472d9-f4f5-432d-b53d-f4f684c4307b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:53.935568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed54528d-8880-4a74-89bb-41ec0841ed38 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:53.981606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715be672-24f0-44a6-bbea-d39fda44e647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:53.982280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715be672-24f0-44a6-bbea-d39fda44e647 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:53.982571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715be672-24f0-44a6-bbea-d39fda44e647 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:53.982859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715be672-24f0-44a6-bbea-d39fda44e647 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:53.983592 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715be672-24f0-44a6-bbea-d39fda44e647 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:53.983909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715be672-24f0-44a6-bbea-d39fda44e647 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:53.985348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715be672-24f0-44a6-bbea-d39fda44e647 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.001481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 296/591] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:54.002377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715be672-24f0-44a6-bbea-d39fda44e647 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.026366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e472d9-f4f5-432d-b53d-f4f684c4307b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.027353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:54.071342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c726ec1a-bc43-4b10-967c-e89d5ed5c33e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.072401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c726ec1a-bc43-4b10-967c-e89d5ed5c33e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.072679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c726ec1a-bc43-4b10-967c-e89d5ed5c33e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.072976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c726ec1a-bc43-4b10-967c-e89d5ed5c33e None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.075786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:54.083405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c726ec1a-bc43-4b10-967c-e89d5ed5c33e None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.085361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c726ec1a-bc43-4b10-967c-e89d5ed5c33e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c726ec1a-bc43-4b10-967c-e89d5ed5c33e None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:54.104189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:54.105323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jun 10 12:45:54.106462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:54.107310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715be672-24f0-44a6-bbea-d39fda44e647 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.112983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 296/592] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:53 2022] GET /identity/v3/users/neutron => generated 85 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:54.113965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c726ec1a-bc43-4b10-967c-e89d5ed5c33e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.142971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ef2314-c06c-469c-b3e7-31ddd73f1ba5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.143705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ef2314-c06c-469c-b3e7-31ddd73f1ba5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.144084 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ef2314-c06c-469c-b3e7-31ddd73f1ba5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.144464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ef2314-c06c-469c-b3e7-31ddd73f1ba5 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.145493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ef2314-c06c-469c-b3e7-31ddd73f1ba5 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:54.145904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ef2314-c06c-469c-b3e7-31ddd73f1ba5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.147491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51ef2314-c06c-469c-b3e7-31ddd73f1ba5 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.172015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 297/593] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:54.173071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ef2314-c06c-469c-b3e7-31ddd73f1ba5 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.218966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c726ec1a-bc43-4b10-967c-e89d5ed5c33e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.254375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53cf504-7a1e-42b3-baf5-5a88c37c1d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.255403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53cf504-7a1e-42b3-baf5-5a88c37c1d6c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.255827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53cf504-7a1e-42b3-baf5-5a88c37c1d6c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.256233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c53cf504-7a1e-42b3-baf5-5a88c37c1d6c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.263623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53cf504-7a1e-42b3-baf5-5a88c37c1d6c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.265313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c53cf504-7a1e-42b3-baf5-5a88c37c1d6c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.269123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:54.275950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51ef2314-c06c-469c-b3e7-31ddd73f1ba5 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c53cf504-7a1e-42b3-baf5-5a88c37c1d6c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:54.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:54.281851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:54.282922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:54.282922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:54.282922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:54.282922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:54.282922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:54.282922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:54.282922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:54.282922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:54.285116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 297/594] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:54.286150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53cf504-7a1e-42b3-baf5-5a88c37c1d6c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.329435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ee303b-c43f-4d9a-adc6-e073cb0ee72b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.333256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ee303b-c43f-4d9a-adc6-e073cb0ee72b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.333256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ee303b-c43f-4d9a-adc6-e073cb0ee72b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.334231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ee303b-c43f-4d9a-adc6-e073cb0ee72b None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.335397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ee303b-c43f-4d9a-adc6-e073cb0ee72b None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:54.337252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ee303b-c43f-4d9a-adc6-e073cb0ee72b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.338790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ee303b-c43f-4d9a-adc6-e073cb0ee72b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.359281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 298/595] 10.222.0.25 () {56 vars in 1139 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:54.359281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ee303b-c43f-4d9a-adc6-e073cb0ee72b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.373760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:54.420454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c53cf504-7a1e-42b3-baf5-5a88c37c1d6c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.456070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f0fc2a-a46a-4460-a034-aa1f6393336b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.457133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f0fc2a-a46a-4460-a034-aa1f6393336b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.457558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f0fc2a-a46a-4460-a034-aa1f6393336b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.457998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f0fc2a-a46a-4460-a034-aa1f6393336b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.464460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ee303b-c43f-4d9a-adc6-e073cb0ee72b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.469413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f0fc2a-a46a-4460-a034-aa1f6393336b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.471040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f0fc2a-a46a-4460-a034-aa1f6393336b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-60f0fc2a-a46a-4460-a034-aa1f6393336b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:54.488401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:54.489475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:54.490496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:54.490496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:54.490496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:54.490496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:54.492809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 298/596] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:54.493773 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f0fc2a-a46a-4460-a034-aa1f6393336b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.502512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da015e7c-9a9f-4f17-bc5c-3e1579d3e02f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.503295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da015e7c-9a9f-4f17-bc5c-3e1579d3e02f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.504009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da015e7c-9a9f-4f17-bc5c-3e1579d3e02f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.504009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da015e7c-9a9f-4f17-bc5c-3e1579d3e02f None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.504928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da015e7c-9a9f-4f17-bc5c-3e1579d3e02f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:54.505325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da015e7c-9a9f-4f17-bc5c-3e1579d3e02f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.506860 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da015e7c-9a9f-4f17-bc5c-3e1579d3e02f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.521098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:54.524104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 299/597] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:54.525096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da015e7c-9a9f-4f17-bc5c-3e1579d3e02f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.627622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f0fc2a-a46a-4460-a034-aa1f6393336b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.630395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da015e7c-9a9f-4f17-bc5c-3e1579d3e02f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.637589 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:54.666935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429a9fc2-28dd-490a-9f57-9d9c7750ec8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.668862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429a9fc2-28dd-490a-9f57-9d9c7750ec8e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429a9fc2-28dd-490a-9f57-9d9c7750ec8e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.669922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429a9fc2-28dd-490a-9f57-9d9c7750ec8e None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.671142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429a9fc2-28dd-490a-9f57-9d9c7750ec8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:54.671496 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429a9fc2-28dd-490a-9f57-9d9c7750ec8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.672994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429a9fc2-28dd-490a-9f57-9d9c7750ec8e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.674176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2035ee64-a07b-4f7b-b6a1-0442ecc6c818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.675368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2035ee64-a07b-4f7b-b6a1-0442ecc6c818 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.676207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2035ee64-a07b-4f7b-b6a1-0442ecc6c818 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.676631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2035ee64-a07b-4f7b-b6a1-0442ecc6c818 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.686130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 300/598] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:54.690016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2035ee64-a07b-4f7b-b6a1-0442ecc6c818 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.691363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429a9fc2-28dd-490a-9f57-9d9c7750ec8e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.691869 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2035ee64-a07b-4f7b-b6a1-0442ecc6c818 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-2035ee64-a07b-4f7b-b6a1-0442ecc6c818 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:54.716079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:54.717183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:54.718285 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:54.718285 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:54.718285 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:54.718285 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:54.725664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 299/599] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:54.725664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2035ee64-a07b-4f7b-b6a1-0442ecc6c818 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.754963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:54.802391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429a9fc2-28dd-490a-9f57-9d9c7750ec8e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.847844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20716634-d20c-4282-be73-6d70ef22c0f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.849693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20716634-d20c-4282-be73-6d70ef22c0f6 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.850160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20716634-d20c-4282-be73-6d70ef22c0f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.850884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20716634-d20c-4282-be73-6d70ef22c0f6 None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/4a83b4eb626343f3aa5b1c9f3fc828fd/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.855151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2035ee64-a07b-4f7b-b6a1-0442ecc6c818 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:54.876377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20716634-d20c-4282-be73-6d70ef22c0f6 None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=4a83b4eb626343f3aa5b1c9f3fc828fd, role_id=c7c592df9b6b4743a1ba6ca3b2a7fdbc)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.879973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20716634-d20c-4282-be73-6d70ef22c0f6 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.893293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d3193-79a0-412d-ae87-5bb41e161f32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:54.894222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d3193-79a0-412d-ae87-5bb41e161f32 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:54.894617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d3193-79a0-412d-ae87-5bb41e161f32 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:54.895001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6d3193-79a0-412d-ae87-5bb41e161f32 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:54.902481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6d3193-79a0-412d-ae87-5bb41e161f32 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:54.904038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6d3193-79a0-412d-ae87-5bb41e161f32 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-ff6d3193-79a0-412d-ae87-5bb41e161f32 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:54.918476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:54.919574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:54.920602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:54.920602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:54.920602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:54.920602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:54.920602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:54.920602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:54.920602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:54.920602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:54.922388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 300/600] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:54.923596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d3193-79a0-412d-ae87-5bb41e161f32 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:54.944248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:54.958163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 301/601] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:45:54 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/4a83b4eb626343f3aa5b1c9f3fc828fd/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:54.959407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20716634-d20c-4282-be73-6d70ef22c0f6 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:55.088527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6d3193-79a0-412d-ae87-5bb41e161f32 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:55.140111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20716634-d20c-4282-be73-6d70ef22c0f6 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:55.145179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:55.147591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bb655d-58e0-4c44-b630-6a5af60f8a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:55.148271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bb655d-58e0-4c44-b630-6a5af60f8a3b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.149100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bb655d-58e0-4c44-b630-6a5af60f8a3b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.149379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bb655d-58e0-4c44-b630-6a5af60f8a3b None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.150116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bb655d-58e0-4c44-b630-6a5af60f8a3b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:55.150424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bb655d-58e0-4c44-b630-6a5af60f8a3b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:55.154953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0bb655d-58e0-4c44-b630-6a5af60f8a3b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:55.175323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 301/602] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:55.180936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bb655d-58e0-4c44-b630-6a5af60f8a3b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:55.207054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e99cc02-1fec-44b1-be84-6f930e2d8dc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:55.208563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e99cc02-1fec-44b1-be84-6f930e2d8dc6 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.214709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e99cc02-1fec-44b1-be84-6f930e2d8dc6 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.214709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e99cc02-1fec-44b1-be84-6f930e2d8dc6 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.214709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e99cc02-1fec-44b1-be84-6f930e2d8dc6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:55.214709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e99cc02-1fec-44b1-be84-6f930e2d8dc6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:55.214709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e99cc02-1fec-44b1-be84-6f930e2d8dc6 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:55.233467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 302/603] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:55.238369 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e99cc02-1fec-44b1-be84-6f930e2d8dc6 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:55.289912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:55.326194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bb655d-58e0-4c44-b630-6a5af60f8a3b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:55.374929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a77b10-6e2f-4d28-9980-c8c5e0d8953b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:55.376611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a77b10-6e2f-4d28-9980-c8c5e0d8953b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.377294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a77b10-6e2f-4d28-9980-c8c5e0d8953b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.379162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a77b10-6e2f-4d28-9980-c8c5e0d8953b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.388577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a77b10-6e2f-4d28-9980-c8c5e0d8953b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:55.390554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a77b10-6e2f-4d28-9980-c8c5e0d8953b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-72a77b10-6e2f-4d28-9980-c8c5e0d8953b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:55.416522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:55.417506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:55.418383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:55.418383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:55.418383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:55.418383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:55.418383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:55.418383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:55.418383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:55.418383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:55.422971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 302/604] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:55.432420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e99cc02-1fec-44b1-be84-6f930e2d8dc6 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:55.433745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a77b10-6e2f-4d28-9980-c8c5e0d8953b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:55.476113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354b8239-a88a-408f-8619-ad0d77af6493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:55.480431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b8239-a88a-408f-8619-ad0d77af6493 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.481698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b8239-a88a-408f-8619-ad0d77af6493 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.482813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354b8239-a88a-408f-8619-ad0d77af6493 None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/184bf14f33464130ab62ed1853bf4f8d/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.507009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354b8239-a88a-408f-8619-ad0d77af6493 None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=184bf14f33464130ab62ed1853bf4f8d, role_id=c7c592df9b6b4743a1ba6ca3b2a7fdbc)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:55.509524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354b8239-a88a-408f-8619-ad0d77af6493 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:55.572555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a77b10-6e2f-4d28-9980-c8c5e0d8953b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:55.596883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 303/605] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:45:55 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/184bf14f33464130ab62ed1853bf4f8d/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 12:45:55.599114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16d5611-75ec-47f5-9919-f0894901da41 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.599653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16d5611-75ec-47f5-9919-f0894901da41 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.600191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16d5611-75ec-47f5-9919-f0894901da41 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.602950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 304/606] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:55 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:55.613546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf001c-9668-4b8a-a765-dcbcf4f7a5e5 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.613857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf001c-9668-4b8a-a765-dcbcf4f7a5e5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.614871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ebf001c-9668-4b8a-a765-dcbcf4f7a5e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.629821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a3b65d-8f92-4b79-9931-2671112eb9d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:55.630571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a3b65d-8f92-4b79-9931-2671112eb9d7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.630862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a3b65d-8f92-4b79-9931-2671112eb9d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.631143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71a3b65d-8f92-4b79-9931-2671112eb9d7 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.632019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a3b65d-8f92-4b79-9931-2671112eb9d7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:55.632335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a3b65d-8f92-4b79-9931-2671112eb9d7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:55.637700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71a3b65d-8f92-4b79-9931-2671112eb9d7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:55.650455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-4ebf001c-9668-4b8a-a765-dcbcf4f7a5e5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:55.652834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4ebf001c-9668-4b8a-a765-dcbcf4f7a5e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:55.653577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:55.668143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 303/607] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:55.670375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:55.673523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a3b65d-8f92-4b79-9931-2671112eb9d7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:55.793914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 305/608] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:45:55.795329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39212b41-eadf-4b71-a5f7-9d081295d25e None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.795623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39212b41-eadf-4b71-a5f7-9d081295d25e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.795906 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39212b41-eadf-4b71-a5f7-9d081295d25e None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.797555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 306/609] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:55.807817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb15e443-0fd2-4286-b73c-be38f8f9e96a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.808203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb15e443-0fd2-4286-b73c-be38f8f9e96a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.808560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb15e443-0fd2-4286-b73c-be38f8f9e96a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.831176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-fb15e443-0fd2-4286-b73c-be38f8f9e96a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:55.832373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fb15e443-0fd2-4286-b73c-be38f8f9e96a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:55.837968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71a3b65d-8f92-4b79-9931-2671112eb9d7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:55.873974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea41747b-2d77-494c-aa01-82ae16bab9ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:55.874979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea41747b-2d77-494c-aa01-82ae16bab9ae None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:55.875332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea41747b-2d77-494c-aa01-82ae16bab9ae None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:55.875679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea41747b-2d77-494c-aa01-82ae16bab9ae None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/d6794257c90e40bca2c8efd31b0c791a/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:55.891559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea41747b-2d77-494c-aa01-82ae16bab9ae None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=d6794257c90e40bca2c8efd31b0c791a, role_id=c7c592df9b6b4743a1ba6ca3b2a7fdbc)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:55.893548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea41747b-2d77-494c-aa01-82ae16bab9ae None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:55.931585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 307/610] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:55.932656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb15e443-0fd2-4286-b73c-be38f8f9e96a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:55.970877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 304/611] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:45:55 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/d6794257c90e40bca2c8efd31b0c791a/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 12:45:55.975298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea41747b-2d77-494c-aa01-82ae16bab9ae None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:56.064161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:56.091477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:56.121664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb15e443-0fd2-4286-b73c-be38f8f9e96a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:56.185204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19988f23-faa8-4417-9a76-454f919ef442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:56.185838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea41747b-2d77-494c-aa01-82ae16bab9ae None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:56.186598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19988f23-faa8-4417-9a76-454f919ef442 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:56.187090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19988f23-faa8-4417-9a76-454f919ef442 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:56.187725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19988f23-faa8-4417-9a76-454f919ef442 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:56.205273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19988f23-faa8-4417-9a76-454f919ef442 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-19988f23-faa8-4417-9a76-454f919ef442 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:56.213733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:56.214649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:56.215499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:56.218358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 308/612] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 286 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:56.224245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19988f23-faa8-4417-9a76-454f919ef442 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:56.232052 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77415ac-452a-4bd9-a12d-fcc6b5e0e470 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:56.232972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77415ac-452a-4bd9-a12d-fcc6b5e0e470 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:56.233265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77415ac-452a-4bd9-a12d-fcc6b5e0e470 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:56.233545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77415ac-452a-4bd9-a12d-fcc6b5e0e470 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:56.248010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77415ac-452a-4bd9-a12d-fcc6b5e0e470 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-d77415ac-452a-4bd9-a12d-fcc6b5e0e470 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:56.258945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:56.259956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:56.260833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 305/613] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:56.267774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77415ac-452a-4bd9-a12d-fcc6b5e0e470 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:56.404352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77415ac-452a-4bd9-a12d-fcc6b5e0e470 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:56.414887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19988f23-faa8-4417-9a76-454f919ef442 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:56.435204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:56.446181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850902c5-84cc-4521-8add-6de8c536fab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:56.447244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850902c5-84cc-4521-8add-6de8c536fab7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:56.448940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850902c5-84cc-4521-8add-6de8c536fab7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:56.449421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850902c5-84cc-4521-8add-6de8c536fab7 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:56.456111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850902c5-84cc-4521-8add-6de8c536fab7 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:56.456111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850902c5-84cc-4521-8add-6de8c536fab7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:56.456111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-850902c5-84cc-4521-8add-6de8c536fab7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:56.472645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 306/614] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:56.478610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850902c5-84cc-4521-8add-6de8c536fab7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:56.498542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69433685-6266-4ea4-920c-08f38c8c8653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:56.499228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69433685-6266-4ea4-920c-08f38c8c8653 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:56.499512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69433685-6266-4ea4-920c-08f38c8c8653 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:56.499789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69433685-6266-4ea4-920c-08f38c8c8653 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:56.500543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69433685-6266-4ea4-920c-08f38c8c8653 None admin] RBAC: Adding query filter params (name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:56.500877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69433685-6266-4ea4-920c-08f38c8c8653 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:56.504870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69433685-6266-4ea4-920c-08f38c8c8653 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:56.508463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:56.523159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 309/615] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:56.529532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69433685-6266-4ea4-920c-08f38c8c8653 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:56.654611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-850902c5-84cc-4521-8add-6de8c536fab7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:56.717928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a09d84e-31a4-4aea-87ae-e16558a580f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:56.718726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09d84e-31a4-4aea-87ae-e16558a580f6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:56.719329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09d84e-31a4-4aea-87ae-e16558a580f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:56.719605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a09d84e-31a4-4aea-87ae-e16558a580f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:56.741380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69433685-6266-4ea4-920c-08f38c8c8653 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:56.744199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a09d84e-31a4-4aea-87ae-e16558a580f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:56.745766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a09d84e-31a4-4aea-87ae-e16558a580f6 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-1a09d84e-31a4-4aea-87ae-e16558a580f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:56.781720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:56.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:56.783545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:56.783545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:56.783545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:56.783545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:56.783545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 307/616] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:56.787345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2010f56b-8ce8-48d4-880c-723a31d95176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:56.788372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2010f56b-8ce8-48d4-880c-723a31d95176 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:56.788927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2010f56b-8ce8-48d4-880c-723a31d95176 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:56.789448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2010f56b-8ce8-48d4-880c-723a31d95176 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:56.789735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a09d84e-31a4-4aea-87ae-e16558a580f6 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:56.800802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2010f56b-8ce8-48d4-880c-723a31d95176 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:56.802528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2010f56b-8ce8-48d4-880c-723a31d95176 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-2010f56b-8ce8-48d4-880c-723a31d95176 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:56.821268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:56.822206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:56.823083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:56.823083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:56.823083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:56.823083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:56.825657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 310/617] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:45:56.830549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2010f56b-8ce8-48d4-880c-723a31d95176 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:56.854752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:56.905241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:56.970387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2010f56b-8ce8-48d4-880c-723a31d95176 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:56.995320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a09d84e-31a4-4aea-87ae-e16558a580f6 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:57.034895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83939999-7756-4e57-af47-3f1632f48455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:57.035626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83939999-7756-4e57-af47-3f1632f48455 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.035911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83939999-7756-4e57-af47-3f1632f48455 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.036178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83939999-7756-4e57-af47-3f1632f48455 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.040577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83939999-7756-4e57-af47-3f1632f48455 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:57.040577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83939999-7756-4e57-af47-3f1632f48455 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:57.040577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83939999-7756-4e57-af47-3f1632f48455 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:57.048045 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2eeee-b803-46d1-8b77-7ec0b517eaf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:57.048958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2eeee-b803-46d1-8b77-7ec0b517eaf7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.049386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2eeee-b803-46d1-8b77-7ec0b517eaf7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.049899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2eeee-b803-46d1-8b77-7ec0b517eaf7 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.050767 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b2eeee-b803-46d1-8b77-7ec0b517eaf7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:57.051217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b2eeee-b803-46d1-8b77-7ec0b517eaf7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:57.054370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b2eeee-b803-46d1-8b77-7ec0b517eaf7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:57.065082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 311/618] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:57.066127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83939999-7756-4e57-af47-3f1632f48455 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:57.082591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 308/619] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:57.089280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2eeee-b803-46d1-8b77-7ec0b517eaf7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:57.203331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83939999-7756-4e57-af47-3f1632f48455 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:57.212711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:57.237930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f115cf11-9e56-4340-8fb1-9bc3ae3b5c86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:57.238824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f115cf11-9e56-4340-8fb1-9bc3ae3b5c86 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.239200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f115cf11-9e56-4340-8fb1-9bc3ae3b5c86 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.239554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f115cf11-9e56-4340-8fb1-9bc3ae3b5c86 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.248259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f115cf11-9e56-4340-8fb1-9bc3ae3b5c86 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:57.249851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f115cf11-9e56-4340-8fb1-9bc3ae3b5c86 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:57.253844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-f115cf11-9e56-4340-8fb1-9bc3ae3b5c86 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:57.265454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:57.266966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:57.267908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 312/620] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Jun 10 12:45:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:57.274698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f115cf11-9e56-4340-8fb1-9bc3ae3b5c86 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:57.308812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2eeee-b803-46d1-8b77-7ec0b517eaf7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:57.384169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe35a63c-68a2-49c3-b5d4-9ee17891a3c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:57.385007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe35a63c-68a2-49c3-b5d4-9ee17891a3c8 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.385291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe35a63c-68a2-49c3-b5d4-9ee17891a3c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.385564 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe35a63c-68a2-49c3-b5d4-9ee17891a3c8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.400117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe35a63c-68a2-49c3-b5d4-9ee17891a3c8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:57.401628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe35a63c-68a2-49c3-b5d4-9ee17891a3c8 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-fe35a63c-68a2-49c3-b5d4-9ee17891a3c8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:57.418390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:57.419390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:57.420336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 309/621] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:45:57 2022] GET /identity/v3/users/swift => generated 83 bytes in 332 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:57.425361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe35a63c-68a2-49c3-b5d4-9ee17891a3c8 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:57.453191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f115cf11-9e56-4340-8fb1-9bc3ae3b5c86 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:57.491091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f050f1c-8db7-45c3-badf-6b1ffcba5906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:57.491676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f050f1c-8db7-45c3-badf-6b1ffcba5906 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.491976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f050f1c-8db7-45c3-badf-6b1ffcba5906 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.492258 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f050f1c-8db7-45c3-badf-6b1ffcba5906 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.493131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f050f1c-8db7-45c3-badf-6b1ffcba5906 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:57.493441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f050f1c-8db7-45c3-badf-6b1ffcba5906 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:57.495032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f050f1c-8db7-45c3-badf-6b1ffcba5906 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:57.528536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 313/622] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Jun 10 12:45:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:57.530930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba70176-6624-4f79-8ceb-7c1446113aa8 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.531432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba70176-6624-4f79-8ceb-7c1446113aa8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.531932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba70176-6624-4f79-8ceb-7c1446113aa8 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.534606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 314/623] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:57.535537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba70176-6624-4f79-8ceb-7c1446113aa8 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:57.553329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe35a63c-68a2-49c3-b5d4-9ee17891a3c8 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:57.572736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:57.589959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f141a4f0-b00d-4483-ae39-a57fd0c66fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:57.590919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f141a4f0-b00d-4483-ae39-a57fd0c66fc3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.591396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f141a4f0-b00d-4483-ae39-a57fd0c66fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.591679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f141a4f0-b00d-4483-ae39-a57fd0c66fc3 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.592592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f141a4f0-b00d-4483-ae39-a57fd0c66fc3 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:57.592939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f141a4f0-b00d-4483-ae39-a57fd0c66fc3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:57.594379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f141a4f0-b00d-4483-ae39-a57fd0c66fc3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:57.610722 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:57.618685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 310/624] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Jun 10 12:45:57 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:45:57.618685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc90094e-b899-4cbf-b941-acea879766e0 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.618685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc90094e-b899-4cbf-b941-acea879766e0 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.618685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc90094e-b899-4cbf-b941-acea879766e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.636822 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-bc90094e-b899-4cbf-b941-acea879766e0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:57.638120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-bc90094e-b899-4cbf-b941-acea879766e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:57.675058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba70176-6624-4f79-8ceb-7c1446113aa8 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:57.719721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1c8261-a06a-4dba-9a8a-93906f16e5f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:57.720373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c8261-a06a-4dba-9a8a-93906f16e5f1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.720660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c8261-a06a-4dba-9a8a-93906f16e5f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.720974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1c8261-a06a-4dba-9a8a-93906f16e5f1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.725078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 311/625] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:57.726054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc90094e-b899-4cbf-b941-acea879766e0 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:57.797577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1c8261-a06a-4dba-9a8a-93906f16e5f1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:57.799078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1c8261-a06a-4dba-9a8a-93906f16e5f1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:57.829012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:57.848732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc90094e-b899-4cbf-b941-acea879766e0 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:57.879919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:57.887182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb798a2-f721-40c5-a504-c6849a3574dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:57.888258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb798a2-f721-40c5-a504-c6849a3574dd None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:57.888655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb798a2-f721-40c5-a504-c6849a3574dd None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:57.889068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb798a2-f721-40c5-a504-c6849a3574dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:57.900487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb798a2-f721-40c5-a504-c6849a3574dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:57.902576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb798a2-f721-40c5-a504-c6849a3574dd None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:57.909547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 315/626] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:45:57.910411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1c8261-a06a-4dba-9a8a-93906f16e5f1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-dcb798a2-f721-40c5-a504-c6849a3574dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:57.927502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:57.928471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:57.929401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:57.929401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:57.929401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:57.929401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:57.929401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 312/627] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:57.931257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb798a2-f721-40c5-a504-c6849a3574dd None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.019448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1c8261-a06a-4dba-9a8a-93906f16e5f1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.045506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb798a2-f721-40c5-a504-c6849a3574dd None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.048453 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:58.057790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206dbdce-a88a-4806-ac28-e572e48d7081 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.058665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206dbdce-a88a-4806-ac28-e572e48d7081 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.058876 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206dbdce-a88a-4806-ac28-e572e48d7081 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.059156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206dbdce-a88a-4806-ac28-e572e48d7081 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.069300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206dbdce-a88a-4806-ac28-e572e48d7081 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-206dbdce-a88a-4806-ac28-e572e48d7081 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:58.077115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:58.078039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:58.078940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 316/628] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:58.080223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206dbdce-a88a-4806-ac28-e572e48d7081 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.086405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ff4c0bc-0c4f-4ab0-ab45-1285a08c6b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.087197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff4c0bc-0c4f-4ab0-ab45-1285a08c6b56 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.087480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff4c0bc-0c4f-4ab0-ab45-1285a08c6b56 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.087754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ff4c0bc-0c4f-4ab0-ab45-1285a08c6b56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.099014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff4c0bc-0c4f-4ab0-ab45-1285a08c6b56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:58.100487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ff4c0bc-0c4f-4ab0-ab45-1285a08c6b56 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-8ff4c0bc-0c4f-4ab0-ab45-1285a08c6b56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:58.118408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:58.119362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:58.120251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:58.120251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:58.120251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:58.120251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:58.120251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 313/629] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:58.124815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8ac96d-9333-4547-b275-b7225dc67d58 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.124815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8ac96d-9333-4547-b275-b7225dc67d58 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.124815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8ac96d-9333-4547-b275-b7225dc67d58 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.124815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 314/630] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:58.124815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8ac96d-9333-4547-b275-b7225dc67d58 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.163645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:58.190415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206dbdce-a88a-4806-ac28-e572e48d7081 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.226926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c77071c-554b-4303-9e6e-d571b15798a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.227591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c77071c-554b-4303-9e6e-d571b15798a0 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.227876 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c77071c-554b-4303-9e6e-d571b15798a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.228156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c77071c-554b-4303-9e6e-d571b15798a0 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.228907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c77071c-554b-4303-9e6e-d571b15798a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:58.229218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c77071c-554b-4303-9e6e-d571b15798a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:58.230604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c77071c-554b-4303-9e6e-d571b15798a0 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:58.238526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8ac96d-9333-4547-b275-b7225dc67d58 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.248141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 317/631] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:58.248141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c77071c-554b-4303-9e6e-d571b15798a0 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.281945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-055c5fbd-d254-4da9-8b1f-1f02c1cd8303 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.282934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055c5fbd-d254-4da9-8b1f-1f02c1cd8303 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.283253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055c5fbd-d254-4da9-8b1f-1f02c1cd8303 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.284357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055c5fbd-d254-4da9-8b1f-1f02c1cd8303 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.285153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055c5fbd-d254-4da9-8b1f-1f02c1cd8303 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:58.285468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055c5fbd-d254-4da9-8b1f-1f02c1cd8303 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:58.286853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-055c5fbd-d254-4da9-8b1f-1f02c1cd8303 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:58.296353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:58.304065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 315/632] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:58.304065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5041a6c7-7c95-420a-9af5-cd7fa59e1979 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.304065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5041a6c7-7c95-420a-9af5-cd7fa59e1979 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.304065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5041a6c7-7c95-420a-9af5-cd7fa59e1979 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.319197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5041a6c7-7c95-420a-9af5-cd7fa59e1979 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:58.320413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5041a6c7-7c95-420a-9af5-cd7fa59e1979 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:58.357000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c77071c-554b-4303-9e6e-d571b15798a0 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.392000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd4b0e5-915b-4d35-9bae-bef7b498078d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.392674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd4b0e5-915b-4d35-9bae-bef7b498078d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.392969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd4b0e5-915b-4d35-9bae-bef7b498078d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.393301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd4b0e5-915b-4d35-9bae-bef7b498078d None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.393467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 316/633] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:58.394117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd4b0e5-915b-4d35-9bae-bef7b498078d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:58.394431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd4b0e5-915b-4d35-9bae-bef7b498078d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:58.394761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5041a6c7-7c95-420a-9af5-cd7fa59e1979 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.395848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd4b0e5-915b-4d35-9bae-bef7b498078d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:58.421683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 318/634] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:58.422841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd4b0e5-915b-4d35-9bae-bef7b498078d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.440546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:58.512895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5041a6c7-7c95-420a-9af5-cd7fa59e1979 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.521851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd4b0e5-915b-4d35-9bae-bef7b498078d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.528659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:58.546027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2029b027-0a1b-432b-8b37-dcb474d1893a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.546811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2029b027-0a1b-432b-8b37-dcb474d1893a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.547092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2029b027-0a1b-432b-8b37-dcb474d1893a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.547367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2029b027-0a1b-432b-8b37-dcb474d1893a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.553706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2029b027-0a1b-432b-8b37-dcb474d1893a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:58.554918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bb292-fdcf-4c2e-bc8c-5afe036244f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.555216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2029b027-0a1b-432b-8b37-dcb474d1893a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:58.555637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bb292-fdcf-4c2e-bc8c-5afe036244f1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.555916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bb292-fdcf-4c2e-bc8c-5afe036244f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.556186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628bb292-fdcf-4c2e-bc8c-5afe036244f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.566719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628bb292-fdcf-4c2e-bc8c-5afe036244f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:58.568141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628bb292-fdcf-4c2e-bc8c-5afe036244f1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-2029b027-0a1b-432b-8b37-dcb474d1893a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:58.568324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:58.569365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:58.570972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:58.570972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:58.570972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:58.570972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:58.570972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:58.570972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:58.570972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:58.570972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:58.570972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 317/635] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:58.571578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2029b027-0a1b-432b-8b37-dcb474d1893a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-628bb292-fdcf-4c2e-bc8c-5afe036244f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:58.590023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:58.591095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:58.592018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:58.592018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:58.592018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:58.592018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:58.592018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 319/636] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:58.592326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bb292-fdcf-4c2e-bc8c-5afe036244f1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.656867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:58.671187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2029b027-0a1b-432b-8b37-dcb474d1893a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.689218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628bb292-fdcf-4c2e-bc8c-5afe036244f1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.704842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8552cc-3ac9-41e5-9e00-9409c7ed9132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.705623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8552cc-3ac9-41e5-9e00-9409c7ed9132 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.705910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8552cc-3ac9-41e5-9e00-9409c7ed9132 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.706178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8552cc-3ac9-41e5-9e00-9409c7ed9132 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.715874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8552cc-3ac9-41e5-9e00-9409c7ed9132 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:58.722013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03939db0-d9dc-4047-95ff-a832d6fc851b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.722792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03939db0-d9dc-4047-95ff-a832d6fc851b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.723079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03939db0-d9dc-4047-95ff-a832d6fc851b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-4a8552cc-3ac9-41e5-9e00-9409c7ed9132 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:58.723340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:58.724395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:58.725707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:58.726483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03939db0-d9dc-4047-95ff-a832d6fc851b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.726559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 318/637] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:58.726559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dec890-eb98-48d6-a3fe-c85f08cc5879 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.726559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dec890-eb98-48d6-a3fe-c85f08cc5879 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.726559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53dec890-eb98-48d6-a3fe-c85f08cc5879 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.727765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 319/638] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:45:58 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:45:58.732171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dec890-eb98-48d6-a3fe-c85f08cc5879 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.732651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03939db0-d9dc-4047-95ff-a832d6fc851b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:58.734177 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03939db0-d9dc-4047-95ff-a832d6fc851b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-03939db0-d9dc-4047-95ff-a832d6fc851b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:45:58.749796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:58.750883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 320/639] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:58.751784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03939db0-d9dc-4047-95ff-a832d6fc851b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.774974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:58.838967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53dec890-eb98-48d6-a3fe-c85f08cc5879 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.853492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03939db0-d9dc-4047-95ff-a832d6fc851b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.871129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fe8e9f-05a3-488d-ab4e-b7288bbe709d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.871787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe8e9f-05a3-488d-ab4e-b7288bbe709d None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.872071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe8e9f-05a3-488d-ab4e-b7288bbe709d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.872350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51fe8e9f-05a3-488d-ab4e-b7288bbe709d None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.873176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe8e9f-05a3-488d-ab4e-b7288bbe709d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:58.873472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe8e9f-05a3-488d-ab4e-b7288bbe709d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:58.874904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51fe8e9f-05a3-488d-ab4e-b7288bbe709d None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:58.887136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d1f9d9-2a00-4269-a2e9-48669b629c17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:58.887827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d1f9d9-2a00-4269-a2e9-48669b629c17 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.888122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d1f9d9-2a00-4269-a2e9-48669b629c17 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.888424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d1f9d9-2a00-4269-a2e9-48669b629c17 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.889111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d1f9d9-2a00-4269-a2e9-48669b629c17 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:58.889415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d1f9d9-2a00-4269-a2e9-48669b629c17 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:58.890711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 320/640] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:58.890887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d1f9d9-2a00-4269-a2e9-48669b629c17 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:58.891531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fe8e9f-05a3-488d-ab4e-b7288bbe709d None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:58.908684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 321/641] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:58.909047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:58.909157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75cf822-fbc7-49a9-ba66-e7d65f918a90 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:58.909157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75cf822-fbc7-49a9-ba66-e7d65f918a90 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:58.909157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75cf822-fbc7-49a9-ba66-e7d65f918a90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:58.923238 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e75cf822-fbc7-49a9-ba66-e7d65f918a90 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:45:58.929139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e75cf822-fbc7-49a9-ba66-e7d65f918a90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:45:58.992747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 322/642] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:45:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:45:58.993312 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51fe8e9f-05a3-488d-ab4e-b7288bbe709d None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:58.993599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75cf822-fbc7-49a9-ba66-e7d65f918a90 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.003860 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:59.028292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d29fc87-3c68-4e7a-8173-7e988ee4c738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.028977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d29fc87-3c68-4e7a-8173-7e988ee4c738 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.029261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d29fc87-3c68-4e7a-8173-7e988ee4c738 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.029535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d29fc87-3c68-4e7a-8173-7e988ee4c738 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.030269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d29fc87-3c68-4e7a-8173-7e988ee4c738 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:59.030571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d29fc87-3c68-4e7a-8173-7e988ee4c738 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:59.031937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d29fc87-3c68-4e7a-8173-7e988ee4c738 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:59.039716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 321/643] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:59.040619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d29fc87-3c68-4e7a-8173-7e988ee4c738 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.090103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75cf822-fbc7-49a9-ba66-e7d65f918a90 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:59.121469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88f859b-121f-46de-bae4-8518acbef0af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.122147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88f859b-121f-46de-bae4-8518acbef0af None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.122440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88f859b-121f-46de-bae4-8518acbef0af None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.122707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88f859b-121f-46de-bae4-8518acbef0af None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.123475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88f859b-121f-46de-bae4-8518acbef0af None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:59.123780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88f859b-121f-46de-bae4-8518acbef0af None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:59.125254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88f859b-121f-46de-bae4-8518acbef0af None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:59.133948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d29fc87-3c68-4e7a-8173-7e988ee4c738 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:59.137047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 323/644] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:45:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:59.137849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88f859b-121f-46de-bae4-8518acbef0af None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.166161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c762095-f6a1-4ec0-8253-8a1c6f9c7b23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.167030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c762095-f6a1-4ec0-8253-8a1c6f9c7b23 None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.167311 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c762095-f6a1-4ec0-8253-8a1c6f9c7b23 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.167582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c762095-f6a1-4ec0-8253-8a1c6f9c7b23 None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/bb18cd41c9a14d358aac14ed62ef9e02/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.182232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c762095-f6a1-4ec0-8253-8a1c6f9c7b23 None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=bb18cd41c9a14d358aac14ed62ef9e02, role_id=c7c592df9b6b4743a1ba6ca3b2a7fdbc)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:59.183659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c762095-f6a1-4ec0-8253-8a1c6f9c7b23 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:59.233011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88f859b-121f-46de-bae4-8518acbef0af None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:59.245548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:59.265128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf5a2c9-3fea-43f6-8a4e-e63e09e4a99c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.265917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf5a2c9-3fea-43f6-8a4e-e63e09e4a99c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.266203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf5a2c9-3fea-43f6-8a4e-e63e09e4a99c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.266480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf5a2c9-3fea-43f6-8a4e-e63e09e4a99c None admin] PATH_INFO: `/v3/users/glance` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.276270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf5a2c9-3fea-43f6-8a4e-e63e09e4a99c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:59.278015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf5a2c9-3fea-43f6-8a4e-e63e09e4a99c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-daf5a2c9-3fea-43f6-8a4e-e63e09e4a99c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:45:59.294852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:59.296065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 324/645] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:45:59 2022] GET /identity/v3/users/glance => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:59.297504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf5a2c9-3fea-43f6-8a4e-e63e09e4a99c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.314635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 322/646] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:45:59 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/bb18cd41c9a14d358aac14ed62ef9e02/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:45:59.315694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c762095-f6a1-4ec0-8253-8a1c6f9c7b23 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.352462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:59.420266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf5a2c9-3fea-43f6-8a4e-e63e09e4a99c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:59.449758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c762095-f6a1-4ec0-8253-8a1c6f9c7b23 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:59.450288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:59.460208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ee301d-4656-43d3-b4aa-bce99b099a0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.466230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ee301d-4656-43d3-b4aa-bce99b099a0b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.466230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ee301d-4656-43d3-b4aa-bce99b099a0b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.466230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ee301d-4656-43d3-b4aa-bce99b099a0b None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.469419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:59.474502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4ee301d-4656-43d3-b4aa-bce99b099a0b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-e4ee301d-4656-43d3-b4aa-bce99b099a0b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:45:59.482614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:59.483588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 325/647] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:45:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:59.484662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ee301d-4656-43d3-b4aa-bce99b099a0b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.494516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff93f814-d8dd-4804-be41-9946188dfb56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.495307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff93f814-d8dd-4804-be41-9946188dfb56 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.495588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff93f814-d8dd-4804-be41-9946188dfb56 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.495861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff93f814-d8dd-4804-be41-9946188dfb56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.506686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff93f814-d8dd-4804-be41-9946188dfb56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:59.508231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff93f814-d8dd-4804-be41-9946188dfb56 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-ff93f814-d8dd-4804-be41-9946188dfb56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:45:59.529947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:45:59.531086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:45:59.532161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:45:59.532161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:45:59.532161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:45:59.532161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:45:59.533804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 323/648] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:45:59.534793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff93f814-d8dd-4804-be41-9946188dfb56 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.593204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4ee301d-4656-43d3-b4aa-bce99b099a0b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:59.627133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5950a505-de3f-4273-9df8-de2c8b982242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.627708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5950a505-de3f-4273-9df8-de2c8b982242 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.627990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5950a505-de3f-4273-9df8-de2c8b982242 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.628267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5950a505-de3f-4273-9df8-de2c8b982242 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.629117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5950a505-de3f-4273-9df8-de2c8b982242 None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=e9a0496199134369a0ef740c099eed59) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:59.629427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5950a505-de3f-4273-9df8-de2c8b982242 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:59.631283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5950a505-de3f-4273-9df8-de2c8b982242 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:59.637464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 326/649] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:45:59 2022] GET /identity/v3/role_assignments?user.id=e9a0496199134369a0ef740c099eed59&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:59.638330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5950a505-de3f-4273-9df8-de2c8b982242 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.652964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff93f814-d8dd-4804-be41-9946188dfb56 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:59.698207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42f0d49-ea9e-43c0-af5e-a7c27ddedace None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.699154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f0d49-ea9e-43c0-af5e-a7c27ddedace None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.699613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f0d49-ea9e-43c0-af5e-a7c27ddedace None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.700112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42f0d49-ea9e-43c0-af5e-a7c27ddedace None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.701205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f0d49-ea9e-43c0-af5e-a7c27ddedace None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:59.701726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f0d49-ea9e-43c0-af5e-a7c27ddedace None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:59.703380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d42f0d49-ea9e-43c0-af5e-a7c27ddedace None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:59.713549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:59.725510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 324/650] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Jun 10 12:45:59 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:59.726749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42f0d49-ea9e-43c0-af5e-a7c27ddedace None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.761819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:45:59.782779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5950a505-de3f-4273-9df8-de2c8b982242 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:59.832299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafe5d0e-2902-4300-9a25-2a718891eb20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.833280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafe5d0e-2902-4300-9a25-2a718891eb20 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.833712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafe5d0e-2902-4300-9a25-2a718891eb20 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.834160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafe5d0e-2902-4300-9a25-2a718891eb20 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.835509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafe5d0e-2902-4300-9a25-2a718891eb20 None admin] RBAC: Adding query filter params (name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:59.836487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafe5d0e-2902-4300-9a25-2a718891eb20 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:59.838130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafe5d0e-2902-4300-9a25-2a718891eb20 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:59.843763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42f0d49-ea9e-43c0-af5e-a7c27ddedace None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:45:59.853706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 327/651] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:45:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:59.854642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafe5d0e-2902-4300-9a25-2a718891eb20 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.889507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dad5bb-0e66-4df4-8242-6ce7b2999e2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:45:59.890631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dad5bb-0e66-4df4-8242-6ce7b2999e2e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:45:59.891498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dad5bb-0e66-4df4-8242-6ce7b2999e2e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:45:59.892271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16dad5bb-0e66-4df4-8242-6ce7b2999e2e None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:45:59.894093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dad5bb-0e66-4df4-8242-6ce7b2999e2e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:45:59.894613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dad5bb-0e66-4df4-8242-6ce7b2999e2e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:45:59.896857 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16dad5bb-0e66-4df4-8242-6ce7b2999e2e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:45:59.912251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 325/652] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:45:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:45:59.917259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dad5bb-0e66-4df4-8242-6ce7b2999e2e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:45:59.993400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafe5d0e-2902-4300-9a25-2a718891eb20 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:00.018046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:00.057465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d0b847-a971-46ff-9a5f-c80f30d5f146 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:00.058131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d0b847-a971-46ff-9a5f-c80f30d5f146 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:00.058416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d0b847-a971-46ff-9a5f-c80f30d5f146 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:00.058691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d0b847-a971-46ff-9a5f-c80f30d5f146 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:00.086447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:00.099414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16dad5bb-0e66-4df4-8242-6ce7b2999e2e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:00.143973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b91187-6fc4-4801-b0ee-19b96b47877d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:00.144794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b91187-6fc4-4801-b0ee-19b96b47877d None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:00.144987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b91187-6fc4-4801-b0ee-19b96b47877d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:00.145289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b91187-6fc4-4801-b0ee-19b96b47877d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:00.156503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b91187-6fc4-4801-b0ee-19b96b47877d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:00.157991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b91187-6fc4-4801-b0ee-19b96b47877d None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-23b91187-6fc4-4801-b0ee-19b96b47877d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:00.177267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:00.178248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:00.179247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:00.179247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:00.179247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:00.179247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:00.179247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 326/653] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:45:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:00.180637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b91187-6fc4-4801-b0ee-19b96b47877d None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:00.201913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d0b847-a971-46ff-9a5f-c80f30d5f146 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:00.204038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d0b847-a971-46ff-9a5f-c80f30d5f146 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:00.310331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b91187-6fc4-4801-b0ee-19b96b47877d None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:00.326349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 328/654] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:45:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:46:00.327796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d0b847-a971-46ff-9a5f-c80f30d5f146 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:00.345756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:00.406308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c62c04d-d9e6-492f-bc0b-3b6165ac0496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:00.406308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c62c04d-d9e6-492f-bc0b-3b6165ac0496 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:00.406308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c62c04d-d9e6-492f-bc0b-3b6165ac0496 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:00.406651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c62c04d-d9e6-492f-bc0b-3b6165ac0496 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:00.428193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c62c04d-d9e6-492f-bc0b-3b6165ac0496 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:00.432757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c62c04d-d9e6-492f-bc0b-3b6165ac0496 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:00.442413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:00.444467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d0b847-a971-46ff-9a5f-c80f30d5f146 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-7c62c04d-d9e6-492f-bc0b-3b6165ac0496 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:00.467546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:00.468734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jun 10 12:46:00.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:00.475922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 327/655] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:00 2022] GET /identity/v3/users/cinder => generated 84 bytes in 296 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:00.476979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c62c04d-d9e6-492f-bc0b-3b6165ac0496 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:00.487095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e7576-0787-46b6-a4de-2883b5d2b889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:00.488184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e7576-0787-46b6-a4de-2883b5d2b889 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:00.488678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e7576-0787-46b6-a4de-2883b5d2b889 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:00.489229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3e7576-0787-46b6-a4de-2883b5d2b889 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:00.490245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3e7576-0787-46b6-a4de-2883b5d2b889 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:00.490827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3e7576-0787-46b6-a4de-2883b5d2b889 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:00.493001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3e7576-0787-46b6-a4de-2883b5d2b889 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:00.506133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 329/656] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:00.507543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e7576-0787-46b6-a4de-2883b5d2b889 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:00.612907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:00.625646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c62c04d-d9e6-492f-bc0b-3b6165ac0496 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:00.641827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3e7576-0787-46b6-a4de-2883b5d2b889 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:00.660520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0ef3b2-4c83-48fd-be25-08dc1a138eb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:00.661286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ef3b2-4c83-48fd-be25-08dc1a138eb2 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:00.661534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ef3b2-4c83-48fd-be25-08dc1a138eb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:00.661809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ef3b2-4c83-48fd-be25-08dc1a138eb2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:00.672572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0ef3b2-4c83-48fd-be25-08dc1a138eb2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:00.674165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0ef3b2-4c83-48fd-be25-08dc1a138eb2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:00.681986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9529ff0-3890-4479-94fa-d0bf9bf61b93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:00.685043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9529ff0-3890-4479-94fa-d0bf9bf61b93 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:00.685540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9529ff0-3890-4479-94fa-d0bf9bf61b93 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:00.686024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9529ff0-3890-4479-94fa-d0bf9bf61b93 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:00.687093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9529ff0-3890-4479-94fa-d0bf9bf61b93 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:00.689817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9529ff0-3890-4479-94fa-d0bf9bf61b93 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:00.691490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9529ff0-3890-4479-94fa-d0bf9bf61b93 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-4b0ef3b2-4c83-48fd-be25-08dc1a138eb2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:00.693536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:00.694423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:00.695874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:00.695874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:00.695874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:00.695874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:00.700159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 328/657] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:00.701161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0ef3b2-4c83-48fd-be25-08dc1a138eb2 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:00.715586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 330/658] 10.222.0.25 () {56 vars in 1137 bytes} [Fri Jun 10 12:46:00 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:00.716791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9529ff0-3890-4479-94fa-d0bf9bf61b93 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:00.760226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:00.811969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0ef3b2-4c83-48fd-be25-08dc1a138eb2 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:00.844709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9618dd1-c2f8-47d7-8d87-2f39b7d8ed5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:00.845131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9529ff0-3890-4479-94fa-d0bf9bf61b93 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:00.845713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9618dd1-c2f8-47d7-8d87-2f39b7d8ed5e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:00.846002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9618dd1-c2f8-47d7-8d87-2f39b7d8ed5e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:00.846277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9618dd1-c2f8-47d7-8d87-2f39b7d8ed5e None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:00.854348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9618dd1-c2f8-47d7-8d87-2f39b7d8ed5e None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:00.855603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9618dd1-c2f8-47d7-8d87-2f39b7d8ed5e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-a9618dd1-c2f8-47d7-8d87-2f39b7d8ed5e None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:00.869537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:00.870493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:00.871453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 329/659] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:46:00 2022] GET /identity/v3/users/neutron => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:00.872060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9618dd1-c2f8-47d7-8d87-2f39b7d8ed5e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:00.881653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:00.888871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e08c7d-ed4c-403a-896b-2ad341d38a25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:00.889764 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e08c7d-ed4c-403a-896b-2ad341d38a25 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:00.894830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e08c7d-ed4c-403a-896b-2ad341d38a25 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:00.894830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e08c7d-ed4c-403a-896b-2ad341d38a25 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:00.894830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e08c7d-ed4c-403a-896b-2ad341d38a25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:00.894830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e08c7d-ed4c-403a-896b-2ad341d38a25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:00.894830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e08c7d-ed4c-403a-896b-2ad341d38a25 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:00.907293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 331/660] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:00.908527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e08c7d-ed4c-403a-896b-2ad341d38a25 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:00.978220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9618dd1-c2f8-47d7-8d87-2f39b7d8ed5e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:00.992856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:01.013735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cc0ad2-ffcf-4238-b5f6-d1f47a5c55b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:01.014379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cc0ad2-ffcf-4238-b5f6-d1f47a5c55b7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.014664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cc0ad2-ffcf-4238-b5f6-d1f47a5c55b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.014939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cc0ad2-ffcf-4238-b5f6-d1f47a5c55b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.030329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e08c7d-ed4c-403a-896b-2ad341d38a25 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:01.077087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87545910-62cb-452a-93bf-13a4e6731e6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:01.077910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87545910-62cb-452a-93bf-13a4e6731e6d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.078197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87545910-62cb-452a-93bf-13a4e6731e6d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.078472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87545910-62cb-452a-93bf-13a4e6731e6d None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.079600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87545910-62cb-452a-93bf-13a4e6731e6d None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:01.079913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87545910-62cb-452a-93bf-13a4e6731e6d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:01.081441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87545910-62cb-452a-93bf-13a4e6731e6d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:01.103894 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9cc0ad2-ffcf-4238-b5f6-d1f47a5c55b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:01.104654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 332/661] 10.222.0.25 () {56 vars in 1139 bytes} [Fri Jun 10 12:46:00 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:01.105550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87545910-62cb-452a-93bf-13a4e6731e6d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:01.106334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9cc0ad2-ffcf-4238-b5f6-d1f47a5c55b7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:01.167056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:01.224617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87545910-62cb-452a-93bf-13a4e6731e6d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:01.225904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 330/662] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:46:01.226904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cc0ad2-ffcf-4238-b5f6-d1f47a5c55b7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:01.240780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:01.265197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29363a0c-1f26-4641-aa3e-863e7102b34b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:01.266347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29363a0c-1f26-4641-aa3e-863e7102b34b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.266839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29363a0c-1f26-4641-aa3e-863e7102b34b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.267336 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29363a0c-1f26-4641-aa3e-863e7102b34b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.275959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29363a0c-1f26-4641-aa3e-863e7102b34b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:01.278122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29363a0c-1f26-4641-aa3e-863e7102b34b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-29363a0c-1f26-4641-aa3e-863e7102b34b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:01.293501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:01.294403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:01.295292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:01.295292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:01.295292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:01.295292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:01.295292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:01.295292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:01.295292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:01.295292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:01.297985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 333/663] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:01.298963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29363a0c-1f26-4641-aa3e-863e7102b34b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:01.337900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cc0ad2-ffcf-4238-b5f6-d1f47a5c55b7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:01.383744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e550734-2ed8-4e5c-b4e1-8303154646db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:01.384542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e550734-2ed8-4e5c-b4e1-8303154646db None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.384962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e550734-2ed8-4e5c-b4e1-8303154646db None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.385481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e550734-2ed8-4e5c-b4e1-8303154646db None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.428901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29363a0c-1f26-4641-aa3e-863e7102b34b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:01.442404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:01.470149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e83783a-2aae-40b9-9962-02440e96c299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:01.471670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e83783a-2aae-40b9-9962-02440e96c299 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.472363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e83783a-2aae-40b9-9962-02440e96c299 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.472833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e83783a-2aae-40b9-9962-02440e96c299 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.473401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e550734-2ed8-4e5c-b4e1-8303154646db None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:01.476022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e550734-2ed8-4e5c-b4e1-8303154646db None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:01.489157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e83783a-2aae-40b9-9962-02440e96c299 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:01.491833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e83783a-2aae-40b9-9962-02440e96c299 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:01.498040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-9e83783a-2aae-40b9-9962-02440e96c299 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:01.516700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:01.517677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:01.518582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:01.518582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:01.518582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:01.518582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:01.521470 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 334/664] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:01.522482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e83783a-2aae-40b9-9962-02440e96c299 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:01.607173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 331/665] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:46:01.607849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e550734-2ed8-4e5c-b4e1-8303154646db None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:01.639632 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e83783a-2aae-40b9-9962-02440e96c299 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:01.676037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377a1788-1105-4dd7-9561-57a55781fcc9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:01.677084 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377a1788-1105-4dd7-9561-57a55781fcc9 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.677568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377a1788-1105-4dd7-9561-57a55781fcc9 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.678017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377a1788-1105-4dd7-9561-57a55781fcc9 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.678964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377a1788-1105-4dd7-9561-57a55781fcc9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:01.679507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377a1788-1105-4dd7-9561-57a55781fcc9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:01.681265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377a1788-1105-4dd7-9561-57a55781fcc9 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:01.699694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 335/666] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:01.701172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f9a76d-93d5-436a-8b0d-bb8fb4e05f69 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.701457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f9a76d-93d5-436a-8b0d-bb8fb4e05f69 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.701730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f9a76d-93d5-436a-8b0d-bb8fb4e05f69 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.707149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 336/667] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:01.707149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f9a76d-93d5-436a-8b0d-bb8fb4e05f69 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:01.709841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e550734-2ed8-4e5c-b4e1-8303154646db None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:01.723266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:01.729398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:01.745063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80258ea7-1fec-440c-be6d-ba8e8cf7b02e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:01.745660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80258ea7-1fec-440c-be6d-ba8e8cf7b02e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.745945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80258ea7-1fec-440c-be6d-ba8e8cf7b02e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.746221 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80258ea7-1fec-440c-be6d-ba8e8cf7b02e None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.746947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80258ea7-1fec-440c-be6d-ba8e8cf7b02e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:01.747247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80258ea7-1fec-440c-be6d-ba8e8cf7b02e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:01.748620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80258ea7-1fec-440c-be6d-ba8e8cf7b02e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:01.761187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 332/668] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:01.761902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80258ea7-1fec-440c-be6d-ba8e8cf7b02e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:01.822202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f9a76d-93d5-436a-8b0d-bb8fb4e05f69 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:01.862235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80258ea7-1fec-440c-be6d-ba8e8cf7b02e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:01.866034 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37170d05-d68b-4d6b-9bda-04ea938cdbdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:01.867364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37170d05-d68b-4d6b-9bda-04ea938cdbdf None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.867675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37170d05-d68b-4d6b-9bda-04ea938cdbdf None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.867956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37170d05-d68b-4d6b-9bda-04ea938cdbdf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.878854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37170d05-d68b-4d6b-9bda-04ea938cdbdf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:01.880785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37170d05-d68b-4d6b-9bda-04ea938cdbdf None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:01.897311 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d084b94-ce72-4391-846b-7ad1e317d849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:01.898018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d084b94-ce72-4391-846b-7ad1e317d849 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.898355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d084b94-ce72-4391-846b-7ad1e317d849 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.898632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d084b94-ce72-4391-846b-7ad1e317d849 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.899455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d084b94-ce72-4391-846b-7ad1e317d849 None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=4a83b4eb626343f3aa5b1c9f3fc828fd) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-37170d05-d68b-4d6b-9bda-04ea938cdbdf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:01.899570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:01.900711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:01.901916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:01.901916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:01.901916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:01.901916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:01.902136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d084b94-ce72-4391-846b-7ad1e317d849 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:01.902136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d084b94-ce72-4391-846b-7ad1e317d849 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:01.905427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 337/669] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:01.906597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531d7406-7812-4c0b-8283-6a1ddbef899d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:01.906885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531d7406-7812-4c0b-8283-6a1ddbef899d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:01.907161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531d7406-7812-4c0b-8283-6a1ddbef899d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:01.917368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 333/670] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:01 2022] GET /identity/v3/role_assignments?user.id=4a83b4eb626343f3aa5b1c9f3fc828fd&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:01.917368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d084b94-ce72-4391-846b-7ad1e317d849 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:01.934783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-531d7406-7812-4c0b-8283-6a1ddbef899d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:01.936413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-531d7406-7812-4c0b-8283-6a1ddbef899d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:01.956150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:02.033030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 338/671] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:46:02.033966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531d7406-7812-4c0b-8283-6a1ddbef899d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.038096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:02.043817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d084b94-ce72-4391-846b-7ad1e317d849 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.090765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daaf7f26-0dee-47e4-bf08-c6f2897bae2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.091576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaf7f26-0dee-47e4-bf08-c6f2897bae2f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.091857 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaf7f26-0dee-47e4-bf08-c6f2897bae2f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.092134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daaf7f26-0dee-47e4-bf08-c6f2897bae2f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.099648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daaf7f26-0dee-47e4-bf08-c6f2897bae2f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:02.101701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daaf7f26-0dee-47e4-bf08-c6f2897bae2f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-daaf7f26-0dee-47e4-bf08-c6f2897bae2f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:02.116111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:02.117248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:02.118158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:02.118158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:02.118158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:02.118158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:02.118158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:02.118158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:02.118158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:02.118158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:02.118596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 334/672] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:02.119796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf2098-f666-4ba0-9735-b9c44e18268a None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.120090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf2098-f666-4ba0-9735-b9c44e18268a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.120367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cf2098-f666-4ba0-9735-b9c44e18268a None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.125761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 335/673] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:02.126577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cf2098-f666-4ba0-9735-b9c44e18268a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.160654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531d7406-7812-4c0b-8283-6a1ddbef899d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.199644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679a4128-8e84-460a-b82c-7a14bd5b1f48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.200308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a4128-8e84-460a-b82c-7a14bd5b1f48 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.200588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a4128-8e84-460a-b82c-7a14bd5b1f48 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.200900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679a4128-8e84-460a-b82c-7a14bd5b1f48 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.201616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a4128-8e84-460a-b82c-7a14bd5b1f48 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:02.201924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a4128-8e84-460a-b82c-7a14bd5b1f48 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:02.203292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679a4128-8e84-460a-b82c-7a14bd5b1f48 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:02.214942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 339/674] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:02.216155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679a4128-8e84-460a-b82c-7a14bd5b1f48 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.238042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:02.245521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cf2098-f666-4ba0-9735-b9c44e18268a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.281130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d9a0010-d82b-4eba-a537-6fdc50516601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.281364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:02.282088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9a0010-d82b-4eba-a537-6fdc50516601 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.282442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9a0010-d82b-4eba-a537-6fdc50516601 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.282789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d9a0010-d82b-4eba-a537-6fdc50516601 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.294206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d9a0010-d82b-4eba-a537-6fdc50516601 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-9d9a0010-d82b-4eba-a537-6fdc50516601 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:02.301900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:02.302820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:02.303660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 336/675] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:02.304424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9760416-b488-4253-8d09-0c3f5f45a4b5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.305049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9760416-b488-4253-8d09-0c3f5f45a4b5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.305049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9760416-b488-4253-8d09-0c3f5f45a4b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.323863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a9760416-b488-4253-8d09-0c3f5f45a4b5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:02.325085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a9760416-b488-4253-8d09-0c3f5f45a4b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:02.332240 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679a4128-8e84-460a-b82c-7a14bd5b1f48 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.369712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05929b0d-7433-444e-b558-9c25d2a4c892 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.370365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05929b0d-7433-444e-b558-9c25d2a4c892 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.370657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05929b0d-7433-444e-b558-9c25d2a4c892 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.370937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05929b0d-7433-444e-b558-9c25d2a4c892 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.371720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05929b0d-7433-444e-b558-9c25d2a4c892 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:02.372033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05929b0d-7433-444e-b558-9c25d2a4c892 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:02.373507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05929b0d-7433-444e-b558-9c25d2a4c892 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:02.386042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 340/676] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:02.386858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05929b0d-7433-444e-b558-9c25d2a4c892 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.404223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 337/677] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:46:02.405326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9760416-b488-4253-8d09-0c3f5f45a4b5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.492292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:02.509127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05929b0d-7433-444e-b558-9c25d2a4c892 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.509949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9760416-b488-4253-8d09-0c3f5f45a4b5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.535075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:02.545814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f26ba0d-6169-40af-8968-49773b819d05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.546610 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f26ba0d-6169-40af-8968-49773b819d05 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.546897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f26ba0d-6169-40af-8968-49773b819d05 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.547168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f26ba0d-6169-40af-8968-49773b819d05 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.547278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42960206-f9a2-4bf9-b044-94d6d9162d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.547838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42960206-f9a2-4bf9-b044-94d6d9162d21 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.548139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42960206-f9a2-4bf9-b044-94d6d9162d21 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.548398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42960206-f9a2-4bf9-b044-94d6d9162d21 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.549178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42960206-f9a2-4bf9-b044-94d6d9162d21 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:02.549484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42960206-f9a2-4bf9-b044-94d6d9162d21 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:02.550877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42960206-f9a2-4bf9-b044-94d6d9162d21 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:02.554101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f26ba0d-6169-40af-8968-49773b819d05 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:02.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f26ba0d-6169-40af-8968-49773b819d05 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:02.564416 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 338/678] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:02.564416 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42960206-f9a2-4bf9-b044-94d6d9162d21 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-9f26ba0d-6169-40af-8968-49773b819d05 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:02.573420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:02.574331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 341/679] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:02.575803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f26ba0d-6169-40af-8968-49773b819d05 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.670976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42960206-f9a2-4bf9-b044-94d6d9162d21 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.677338 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f26ba0d-6169-40af-8968-49773b819d05 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.706890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41eee3e0-d797-4613-8cb9-01e06467a50a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.707549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41eee3e0-d797-4613-8cb9-01e06467a50a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.707847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41eee3e0-d797-4613-8cb9-01e06467a50a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.708115 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41eee3e0-d797-4613-8cb9-01e06467a50a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.708981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41eee3e0-d797-4613-8cb9-01e06467a50a None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=184bf14f33464130ab62ed1853bf4f8d) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:02.709295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41eee3e0-d797-4613-8cb9-01e06467a50a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:02.711207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4cb961-cad6-4284-9bb5-7e740f6b1d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.711336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41eee3e0-d797-4613-8cb9-01e06467a50a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:02.711994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4cb961-cad6-4284-9bb5-7e740f6b1d46 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.712271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4cb961-cad6-4284-9bb5-7e740f6b1d46 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.712540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4cb961-cad6-4284-9bb5-7e740f6b1d46 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.729100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 339/680] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/role_assignments?user.id=184bf14f33464130ab62ed1853bf4f8d&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:02.729100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41eee3e0-d797-4613-8cb9-01e06467a50a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.736634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b4cb961-cad6-4284-9bb5-7e740f6b1d46 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:02.743960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-8b4cb961-cad6-4284-9bb5-7e740f6b1d46 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:02.744966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:02.746087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:02.747108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:02.749140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 342/681] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:02.749860 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4cb961-cad6-4284-9bb5-7e740f6b1d46 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.817672 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:02.868868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41eee3e0-d797-4613-8cb9-01e06467a50a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.881448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b4cb961-cad6-4284-9bb5-7e740f6b1d46 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:02.918381 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81920745-fae4-4c3f-8f31-9b356ab31603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.919178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81920745-fae4-4c3f-8f31-9b356ab31603 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.919464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81920745-fae4-4c3f-8f31-9b356ab31603 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.919744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81920745-fae4-4c3f-8f31-9b356ab31603 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.931488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a110a3fc-4d5c-4ec7-aeb0-e5336cf3aede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:02.932151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a110a3fc-4d5c-4ec7-aeb0-e5336cf3aede None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:02.932443 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a110a3fc-4d5c-4ec7-aeb0-e5336cf3aede None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:02.932721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a110a3fc-4d5c-4ec7-aeb0-e5336cf3aede None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:02.933464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81920745-fae4-4c3f-8f31-9b356ab31603 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:02.933574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a110a3fc-4d5c-4ec7-aeb0-e5336cf3aede None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:02.933854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a110a3fc-4d5c-4ec7-aeb0-e5336cf3aede None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:02.935310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a110a3fc-4d5c-4ec7-aeb0-e5336cf3aede None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:02.935635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81920745-fae4-4c3f-8f31-9b356ab31603 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:02.948604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 343/682] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:02.949458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a110a3fc-4d5c-4ec7-aeb0-e5336cf3aede None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-81920745-fae4-4c3f-8f31-9b356ab31603 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:02.956155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:02.957100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:02.957987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:02.957987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:02.957987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:02.957987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:02.964559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 340/683] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:02.964559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81920745-fae4-4c3f-8f31-9b356ab31603 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:03.040564 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:03.076834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a110a3fc-4d5c-4ec7-aeb0-e5336cf3aede None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:03.079880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:03.084631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81920745-fae4-4c3f-8f31-9b356ab31603 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:03.114695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956c2e67-c72a-4487-ba47-3ceb32550d91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:03.115756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956c2e67-c72a-4487-ba47-3ceb32550d91 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:03.116272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956c2e67-c72a-4487-ba47-3ceb32550d91 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:03.116869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956c2e67-c72a-4487-ba47-3ceb32550d91 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:03.117858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956c2e67-c72a-4487-ba47-3ceb32550d91 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:03.118386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956c2e67-c72a-4487-ba47-3ceb32550d91 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:03.120032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956c2e67-c72a-4487-ba47-3ceb32550d91 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:03.127807 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50260576-45a5-431a-b6bb-e3933c0bc1f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:03.128496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50260576-45a5-431a-b6bb-e3933c0bc1f7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:03.128800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50260576-45a5-431a-b6bb-e3933c0bc1f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:03.129092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50260576-45a5-431a-b6bb-e3933c0bc1f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:03.130230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50260576-45a5-431a-b6bb-e3933c0bc1f7 None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=d6794257c90e40bca2c8efd31b0c791a) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:03.130549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50260576-45a5-431a-b6bb-e3933c0bc1f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:03.132006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50260576-45a5-431a-b6bb-e3933c0bc1f7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:03.137321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 344/684] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:03.137321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956c2e67-c72a-4487-ba47-3ceb32550d91 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:03.139117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 341/685] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:02 2022] GET /identity/v3/role_assignments?user.id=d6794257c90e40bca2c8efd31b0c791a&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 608 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:03.140316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50260576-45a5-431a-b6bb-e3933c0bc1f7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:03.258864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50260576-45a5-431a-b6bb-e3933c0bc1f7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:03.293276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:03.301480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8139a779-1fed-4025-a957-149576e3ce77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:03.304899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8139a779-1fed-4025-a957-149576e3ce77 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:03.304899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8139a779-1fed-4025-a957-149576e3ce77 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:03.304899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8139a779-1fed-4025-a957-149576e3ce77 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:03.323327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8139a779-1fed-4025-a957-149576e3ce77 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:03.324030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8139a779-1fed-4025-a957-149576e3ce77 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-8139a779-1fed-4025-a957-149576e3ce77 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:03.372939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:03.373935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:03.379404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:03.379404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:03.379404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:03.379404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:03.379404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 342/686] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:03.384042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8139a779-1fed-4025-a957-149576e3ce77 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:03.406914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956c2e67-c72a-4487-ba47-3ceb32550d91 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:03.450310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921c740-57d2-41e6-934c-78eb89efaed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:03.451362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921c740-57d2-41e6-934c-78eb89efaed7 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:03.452263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921c740-57d2-41e6-934c-78eb89efaed7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:03.452872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6921c740-57d2-41e6-934c-78eb89efaed7 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:03.453828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6921c740-57d2-41e6-934c-78eb89efaed7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:03.454147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6921c740-57d2-41e6-934c-78eb89efaed7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:03.456834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6921c740-57d2-41e6-934c-78eb89efaed7 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:03.470226 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 345/687] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 337 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:03.477778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921c740-57d2-41e6-934c-78eb89efaed7 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:03.509835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8139a779-1fed-4025-a957-149576e3ce77 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:03.543588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:03.548255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d17238-a315-48d5-b8e8-04c0704d796f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:03.549329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d17238-a315-48d5-b8e8-04c0704d796f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:03.549862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d17238-a315-48d5-b8e8-04c0704d796f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:03.550340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d17238-a315-48d5-b8e8-04c0704d796f None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:03.551261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d17238-a315-48d5-b8e8-04c0704d796f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:03.551782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d17238-a315-48d5-b8e8-04c0704d796f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:03.553431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d17238-a315-48d5-b8e8-04c0704d796f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:03.571434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 343/688] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:03.573030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d17238-a315-48d5-b8e8-04c0704d796f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:03.637307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:03.683881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d17238-a315-48d5-b8e8-04c0704d796f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:03.686161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6921c740-57d2-41e6-934c-78eb89efaed7 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:03.734508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10588f9-d9c0-489c-a232-f35ce55ec385 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:03.735665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10588f9-d9c0-489c-a232-f35ce55ec385 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:03.736158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10588f9-d9c0-489c-a232-f35ce55ec385 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:03.736699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10588f9-d9c0-489c-a232-f35ce55ec385 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:03.745324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10588f9-d9c0-489c-a232-f35ce55ec385 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:03.747097 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10588f9-d9c0-489c-a232-f35ce55ec385 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-e10588f9-d9c0-489c-a232-f35ce55ec385 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:03.762290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:03.763284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:03.764219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:03.771161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 344/689] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Jun 10 12:46:03 2022] GET /identity/v3/users/nova => generated 82 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:03.772630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10588f9-d9c0-489c-a232-f35ce55ec385 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:03.829506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30791f71-11ec-4e77-89c5-aacee9252d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:03.830587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30791f71-11ec-4e77-89c5-aacee9252d98 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:03.830587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30791f71-11ec-4e77-89c5-aacee9252d98 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:03.830871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30791f71-11ec-4e77-89c5-aacee9252d98 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:03.844345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30791f71-11ec-4e77-89c5-aacee9252d98 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:03.845746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30791f71-11ec-4e77-89c5-aacee9252d98 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-30791f71-11ec-4e77-89c5-aacee9252d98 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:03.860789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:03.861800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:03.863334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 346/690] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:03 2022] GET /identity/v3/users/swift => generated 83 bytes in 385 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:03.867274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30791f71-11ec-4e77-89c5-aacee9252d98 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:03.883992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:03.887435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10588f9-d9c0-489c-a232-f35ce55ec385 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:03.967699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:03.983723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30791f71-11ec-4e77-89c5-aacee9252d98 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:04.023387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d2f771-2f79-46bd-82b4-c86ed0a6be61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:04.023903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d2f771-2f79-46bd-82b4-c86ed0a6be61 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.024195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d2f771-2f79-46bd-82b4-c86ed0a6be61 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.024473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d2f771-2f79-46bd-82b4-c86ed0a6be61 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.025357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d2f771-2f79-46bd-82b4-c86ed0a6be61 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:04.025677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d2f771-2f79-46bd-82b4-c86ed0a6be61 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:04.027389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d2f771-2f79-46bd-82b4-c86ed0a6be61 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:04.047032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 347/691] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Jun 10 12:46:03 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:04.050501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c7b608-66be-406f-89b1-f01dbf4b75d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:04.051560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c7b608-66be-406f-89b1-f01dbf4b75d1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.052068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c7b608-66be-406f-89b1-f01dbf4b75d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.052367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d2f771-2f79-46bd-82b4-c86ed0a6be61 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:04.052866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c7b608-66be-406f-89b1-f01dbf4b75d1 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.053929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c7b608-66be-406f-89b1-f01dbf4b75d1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:04.054967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c7b608-66be-406f-89b1-f01dbf4b75d1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:04.056688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c7b608-66be-406f-89b1-f01dbf4b75d1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:04.092326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 345/692] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Jun 10 12:46:03 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:04.095463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c7b608-66be-406f-89b1-f01dbf4b75d1 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:04.160005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d2f771-2f79-46bd-82b4-c86ed0a6be61 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:04.196905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7493fa06-0812-4764-9d0b-04247c04b3f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:04.205026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7493fa06-0812-4764-9d0b-04247c04b3f3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.205026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7493fa06-0812-4764-9d0b-04247c04b3f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.205026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7493fa06-0812-4764-9d0b-04247c04b3f3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.231569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:04.297043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c7b608-66be-406f-89b1-f01dbf4b75d1 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:04.324103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7493fa06-0812-4764-9d0b-04247c04b3f3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:04.325953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7493fa06-0812-4764-9d0b-04247c04b3f3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:04.342575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d34b4480-46c6-4f75-91b3-485459239f9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:04.344138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34b4480-46c6-4f75-91b3-485459239f9b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.344620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34b4480-46c6-4f75-91b3-485459239f9b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.345159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34b4480-46c6-4f75-91b3-485459239f9b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.372080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:04.422695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34b4480-46c6-4f75-91b3-485459239f9b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:04.424350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d34b4480-46c6-4f75-91b3-485459239f9b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:04.447015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 348/693] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jun 10 12:46:04.448890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf2390f-2052-43b3-bf58-2cf92392c8df None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.449523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf2390f-2052-43b3-bf58-2cf92392c8df None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.450006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf2390f-2052-43b3-bf58-2cf92392c8df None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.455555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 349/694] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:04.456635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf2390f-2052-43b3-bf58-2cf92392c8df None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:04.507225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:04.540563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 346/695] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 441 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:46:04.540563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42dae9d-2aa9-4b94-ab5d-83ac52453d8a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.540563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42dae9d-2aa9-4b94-ab5d-83ac52453d8a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.540563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d42dae9d-2aa9-4b94-ab5d-83ac52453d8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.560332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-d42dae9d-2aa9-4b94-ab5d-83ac52453d8a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:04.561571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d42dae9d-2aa9-4b94-ab5d-83ac52453d8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:04.575539 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf2390f-2052-43b3-bf58-2cf92392c8df None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:04.615697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcfd854-7203-4c87-8b30-3ce8d12a8b28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:04.617589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcfd854-7203-4c87-8b30-3ce8d12a8b28 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.618090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:04.619229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcfd854-7203-4c87-8b30-3ce8d12a8b28 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.619823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddcfd854-7203-4c87-8b30-3ce8d12a8b28 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.631766 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcfd854-7203-4c87-8b30-3ce8d12a8b28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:04.632306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 347/696] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:46:04.640803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42dae9d-2aa9-4b94-ab5d-83ac52453d8a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:04.642043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddcfd854-7203-4c87-8b30-3ce8d12a8b28 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-ddcfd854-7203-4c87-8b30-3ce8d12a8b28 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:04.667537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:04.668526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:04.670136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:04.670136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:04.670136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:04.670136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:04.670136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 350/697] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:04.674332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcfd854-7203-4c87-8b30-3ce8d12a8b28 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:04.759110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d42dae9d-2aa9-4b94-ab5d-83ac52453d8a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:04.776726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:04.782675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddcfd854-7203-4c87-8b30-3ce8d12a8b28 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:04.798254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ce3758-1bc8-4a8d-9fe3-d20dffe26738 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:04.799054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ce3758-1bc8-4a8d-9fe3-d20dffe26738 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.799337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ce3758-1bc8-4a8d-9fe3-d20dffe26738 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.799614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18ce3758-1bc8-4a8d-9fe3-d20dffe26738 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.810982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ce3758-1bc8-4a8d-9fe3-d20dffe26738 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:04.812452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18ce3758-1bc8-4a8d-9fe3-d20dffe26738 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:04.821197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a218ddc-4967-481f-bf6a-5889b3780401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:04.822223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a218ddc-4967-481f-bf6a-5889b3780401 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.822575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a218ddc-4967-481f-bf6a-5889b3780401 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.822941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a218ddc-4967-481f-bf6a-5889b3780401 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-18ce3758-1bc8-4a8d-9fe3-d20dffe26738 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:04.829617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:04.830681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:04.831729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:04.831729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:04.831729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:04.831729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:04.831729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 348/698] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:04.832035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ce3758-1bc8-4a8d-9fe3-d20dffe26738 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:04.840056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a218ddc-4967-481f-bf6a-5889b3780401 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:04.841867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a218ddc-4967-481f-bf6a-5889b3780401 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-6a218ddc-4967-481f-bf6a-5889b3780401 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:04.862356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:04.863439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:04.864343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:04.864343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:04.864343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:04.864343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:04.865750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 351/699] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:04.866097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a218ddc-4967-481f-bf6a-5889b3780401 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:04.875342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:04.938631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18ce3758-1bc8-4a8d-9fe3-d20dffe26738 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:04.976758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99da5556-7e44-4452-96db-8a1746828f29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:04.977457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99da5556-7e44-4452-96db-8a1746828f29 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:04.977743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99da5556-7e44-4452-96db-8a1746828f29 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:04.978024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99da5556-7e44-4452-96db-8a1746828f29 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:04.978742 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99da5556-7e44-4452-96db-8a1746828f29 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:04.979055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99da5556-7e44-4452-96db-8a1746828f29 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:04.980431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99da5556-7e44-4452-96db-8a1746828f29 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:04.994744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a218ddc-4967-481f-bf6a-5889b3780401 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:04.997064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 349/700] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:04.998003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99da5556-7e44-4452-96db-8a1746828f29 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:05.052450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df7116d-c6e4-4b11-9243-9085450f58d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:05.053154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df7116d-c6e4-4b11-9243-9085450f58d0 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.053486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df7116d-c6e4-4b11-9243-9085450f58d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.053808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df7116d-c6e4-4b11-9243-9085450f58d0 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.054585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df7116d-c6e4-4b11-9243-9085450f58d0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:05.054896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df7116d-c6e4-4b11-9243-9085450f58d0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:05.056351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df7116d-c6e4-4b11-9243-9085450f58d0 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:05.068871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 352/701] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:05.069854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df7116d-c6e4-4b11-9243-9085450f58d0 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:05.081692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:05.137906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:05.141025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99da5556-7e44-4452-96db-8a1746828f29 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:05.180681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a0a7e0-48bf-45fc-9a07-a3070d8a4f5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:05.181400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0a7e0-48bf-45fc-9a07-a3070d8a4f5f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.181683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0a7e0-48bf-45fc-9a07-a3070d8a4f5f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.181965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a0a7e0-48bf-45fc-9a07-a3070d8a4f5f None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.182703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a0a7e0-48bf-45fc-9a07-a3070d8a4f5f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:05.183014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a0a7e0-48bf-45fc-9a07-a3070d8a4f5f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:05.184441 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30a0a7e0-48bf-45fc-9a07-a3070d8a4f5f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:05.197173 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df7116d-c6e4-4b11-9243-9085450f58d0 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:05.201850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 350/702] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:05.202754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a0a7e0-48bf-45fc-9a07-a3070d8a4f5f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:05.242399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0142bca3-a96b-46c5-a1d3-d6be9418ca62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:05.243195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0142bca3-a96b-46c5-a1d3-d6be9418ca62 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.243480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0142bca3-a96b-46c5-a1d3-d6be9418ca62 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.243756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0142bca3-a96b-46c5-a1d3-d6be9418ca62 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.251148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0142bca3-a96b-46c5-a1d3-d6be9418ca62 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:05.252681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0142bca3-a96b-46c5-a1d3-d6be9418ca62 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-0142bca3-a96b-46c5-a1d3-d6be9418ca62 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:05.266852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:05.267759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:05.268671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:05.268671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:05.268671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:05.268671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:05.268671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:05.268671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:05.268671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:05.268671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:05.268671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 353/703] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:05.269822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0142bca3-a96b-46c5-a1d3-d6be9418ca62 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:05.314127 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a0a7e0-48bf-45fc-9a07-a3070d8a4f5f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:05.347458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:05.353964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941fcf9b-74dc-4be8-843a-6a795950fcaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:05.354764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941fcf9b-74dc-4be8-843a-6a795950fcaf None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.355049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941fcf9b-74dc-4be8-843a-6a795950fcaf None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.355325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941fcf9b-74dc-4be8-843a-6a795950fcaf None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.362665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941fcf9b-74dc-4be8-843a-6a795950fcaf None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:05.364111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941fcf9b-74dc-4be8-843a-6a795950fcaf None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-941fcf9b-74dc-4be8-843a-6a795950fcaf None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:05.379415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:05.382886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 351/704] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d234d6-13a4-4e82-83dc-5eb0b99f3055 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.384575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d234d6-13a4-4e82-83dc-5eb0b99f3055 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.385797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d234d6-13a4-4e82-83dc-5eb0b99f3055 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.385797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 352/705] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:05.385797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d234d6-13a4-4e82-83dc-5eb0b99f3055 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:05.400083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0142bca3-a96b-46c5-a1d3-d6be9418ca62 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:05.416849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:05.436826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b79bf2-d2dc-4f49-b0ce-587c04fcf3af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:05.437825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b79bf2-d2dc-4f49-b0ce-587c04fcf3af None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.437825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b79bf2-d2dc-4f49-b0ce-587c04fcf3af None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.438098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b79bf2-d2dc-4f49-b0ce-587c04fcf3af None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.438828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b79bf2-d2dc-4f49-b0ce-587c04fcf3af None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:05.440262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b79bf2-d2dc-4f49-b0ce-587c04fcf3af None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:05.501250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d234d6-13a4-4e82-83dc-5eb0b99f3055 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:05.520229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 354/706] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:46:05 2022] POST /identity/v3/users => generated 254 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:46:05.521551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b79bf2-d2dc-4f49-b0ce-587c04fcf3af None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:05.556534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354af22c-3586-42f9-a6a3-f214ba55bd45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:05.557655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354af22c-3586-42f9-a6a3-f214ba55bd45 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.558162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354af22c-3586-42f9-a6a3-f214ba55bd45 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.558696 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354af22c-3586-42f9-a6a3-f214ba55bd45 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.559787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354af22c-3586-42f9-a6a3-f214ba55bd45 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:05.560306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354af22c-3586-42f9-a6a3-f214ba55bd45 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:05.562236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354af22c-3586-42f9-a6a3-f214ba55bd45 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:05.581345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 353/707] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:05.583475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869fb669-4b8a-4726-8e7f-36f2ada32e83 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.584048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869fb669-4b8a-4726-8e7f-36f2ada32e83 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.584616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869fb669-4b8a-4726-8e7f-36f2ada32e83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.608066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-869fb669-4b8a-4726-8e7f-36f2ada32e83 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:05.609922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-869fb669-4b8a-4726-8e7f-36f2ada32e83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:05.668649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b79bf2-d2dc-4f49-b0ce-587c04fcf3af None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:05.704867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 354/708] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:46:05.706048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869fb669-4b8a-4726-8e7f-36f2ada32e83 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:05.706566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:05.716425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5783ff2-d649-4a0c-8e02-aa50ec480443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:05.717219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5783ff2-d649-4a0c-8e02-aa50ec480443 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.717514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5783ff2-d649-4a0c-8e02-aa50ec480443 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.717804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5783ff2-d649-4a0c-8e02-aa50ec480443 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.719200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5783ff2-d649-4a0c-8e02-aa50ec480443 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:05.719530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5783ff2-d649-4a0c-8e02-aa50ec480443 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:05.722431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5783ff2-d649-4a0c-8e02-aa50ec480443 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:05.725176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:05.737880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 355/709] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:05.739490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5783ff2-d649-4a0c-8e02-aa50ec480443 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:05.833405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869fb669-4b8a-4726-8e7f-36f2ada32e83 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:05.858141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5783ff2-d649-4a0c-8e02-aa50ec480443 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:05.868675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9843e608-16d8-4049-962d-a844842d74f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:05.869721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843e608-16d8-4049-962d-a844842d74f0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.870008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843e608-16d8-4049-962d-a844842d74f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.870281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9843e608-16d8-4049-962d-a844842d74f0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.871109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9843e608-16d8-4049-962d-a844842d74f0 None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=bb18cd41c9a14d358aac14ed62ef9e02) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:05.871419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9843e608-16d8-4049-962d-a844842d74f0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:05.872879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9843e608-16d8-4049-962d-a844842d74f0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:05.884790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 355/710] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:05 2022] GET /identity/v3/role_assignments?user.id=bb18cd41c9a14d358aac14ed62ef9e02&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:05.884790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3709d9c-12ab-4dd3-a970-64aa4f1210a6 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.884790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3709d9c-12ab-4dd3-a970-64aa4f1210a6 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.884790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3709d9c-12ab-4dd3-a970-64aa4f1210a6 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.886610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 356/711] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:05 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:05.887553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3709d9c-12ab-4dd3-a970-64aa4f1210a6 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:05.903543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ad5d2f-9304-4cc2-a281-c44c43842448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:05.904723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad5d2f-9304-4cc2-a281-c44c43842448 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.905172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad5d2f-9304-4cc2-a281-c44c43842448 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.905507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ad5d2f-9304-4cc2-a281-c44c43842448 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.906242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ad5d2f-9304-4cc2-a281-c44c43842448 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:05.907748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83ad5d2f-9304-4cc2-a281-c44c43842448 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-83ad5d2f-9304-4cc2-a281-c44c43842448 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:05.924454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:05.925742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jun 10 12:46:05.926655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:05.927454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 356/712] 10.222.0.25 () {56 vars in 1129 bytes} [Fri Jun 10 12:46:05 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:05.928795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0ec397-52b4-4214-aa91-4e5c7f2f50ca None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:05.929016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0ec397-52b4-4214-aa91-4e5c7f2f50ca None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:05.931632 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0ec397-52b4-4214-aa91-4e5c7f2f50ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:05.947610 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5c0ec397-52b4-4214-aa91-4e5c7f2f50ca None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:05.948954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5c0ec397-52b4-4214-aa91-4e5c7f2f50ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:05.978526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:06.005030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:06.007710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3709d9c-12ab-4dd3-a970-64aa4f1210a6 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:06.040937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 357/713] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:46:06.041934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0ec397-52b4-4214-aa91-4e5c7f2f50ca None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:06.071765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca14cf7-1898-4b58-84a5-1e132e75b1ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:06.072642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca14cf7-1898-4b58-84a5-1e132e75b1ea None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:06.073140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca14cf7-1898-4b58-84a5-1e132e75b1ea None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:06.073423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca14cf7-1898-4b58-84a5-1e132e75b1ea None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:06.074268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca14cf7-1898-4b58-84a5-1e132e75b1ea None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=e9a0496199134369a0ef740c099eed59) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:06.074595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca14cf7-1898-4b58-84a5-1e132e75b1ea None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:06.076179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca14cf7-1898-4b58-84a5-1e132e75b1ea None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:06.082939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 357/714] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:05 2022] GET /identity/v3/role_assignments?user.id=e9a0496199134369a0ef740c099eed59&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=766799d1b6c049168590571f1a959b3b => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:06.083870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca14cf7-1898-4b58-84a5-1e132e75b1ea None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:06.190532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0ec397-52b4-4214-aa91-4e5c7f2f50ca None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:06.215075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca14cf7-1898-4b58-84a5-1e132e75b1ea None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:06.230017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2c0af9-3d69-48c5-b7cd-65ddc922617b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:06.230725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2c0af9-3d69-48c5-b7cd-65ddc922617b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:06.231085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2c0af9-3d69-48c5-b7cd-65ddc922617b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:06.231369 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2c0af9-3d69-48c5-b7cd-65ddc922617b None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:06.232086 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2c0af9-3d69-48c5-b7cd-65ddc922617b None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:06.232399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2c0af9-3d69-48c5-b7cd-65ddc922617b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:06.234832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2c0af9-3d69-48c5-b7cd-65ddc922617b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:06.240508 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 358/715] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Jun 10 12:46:06 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:06.246597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2c0af9-3d69-48c5-b7cd-65ddc922617b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:06.263737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d517520-c678-4a41-9a21-289d4a80092a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:06.264736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d517520-c678-4a41-9a21-289d4a80092a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:06.265052 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d517520-c678-4a41-9a21-289d4a80092a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:06.265768 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d517520-c678-4a41-9a21-289d4a80092a None admin] PATH_INFO: `/v3/services/network` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:06.266727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d517520-c678-4a41-9a21-289d4a80092a None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:06.268305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d517520-c678-4a41-9a21-289d4a80092a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-6d517520-c678-4a41-9a21-289d4a80092a None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:06.285591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:06.286616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:06.287427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 358/716] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:06 2022] GET /identity/v3/services/network => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:06.292846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d517520-c678-4a41-9a21-289d4a80092a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:06.296620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:06.304914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:06.379998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2c0af9-3d69-48c5-b7cd-65ddc922617b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:06.418850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d517520-c678-4a41-9a21-289d4a80092a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:06.428201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e4a8ac-fe4d-4071-a9b2-bf9ee133b3b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:06.429261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e4a8ac-fe4d-4071-a9b2-bf9ee133b3b2 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:06.429699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e4a8ac-fe4d-4071-a9b2-bf9ee133b3b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:06.430149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e4a8ac-fe4d-4071-a9b2-bf9ee133b3b2 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:06.431091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e4a8ac-fe4d-4071-a9b2-bf9ee133b3b2 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:06.431547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e4a8ac-fe4d-4071-a9b2-bf9ee133b3b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:06.433198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e4a8ac-fe4d-4071-a9b2-bf9ee133b3b2 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:06.440156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 359/717] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Jun 10 12:46:06 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:06.459580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a601aa-1619-4c7d-a1dc-413dda6cb51a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:06.460825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a601aa-1619-4c7d-a1dc-413dda6cb51a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:06.461116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a601aa-1619-4c7d-a1dc-413dda6cb51a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:06.461388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a601aa-1619-4c7d-a1dc-413dda6cb51a None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:06.462109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a601aa-1619-4c7d-a1dc-413dda6cb51a None admin] RBAC: Adding query filter params (name=network) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:06.462412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a601aa-1619-4c7d-a1dc-413dda6cb51a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:06.463825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a601aa-1619-4c7d-a1dc-413dda6cb51a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:06.472905 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 359/718] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:06 2022] GET /identity/v3/services?name=network => generated 124 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:06.475286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e4a8ac-fe4d-4071-a9b2-bf9ee133b3b2 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:06.588468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e4a8ac-fe4d-4071-a9b2-bf9ee133b3b2 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:06.596178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:06.625392 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433e9241-067e-46de-8db6-a9d6fd5dd651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:06.626069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433e9241-067e-46de-8db6-a9d6fd5dd651 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:06.626351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433e9241-067e-46de-8db6-a9d6fd5dd651 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:06.626627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433e9241-067e-46de-8db6-a9d6fd5dd651 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:06.627343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433e9241-067e-46de-8db6-a9d6fd5dd651 None admin] RBAC: Adding query filter params (type=network) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:06.627690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433e9241-067e-46de-8db6-a9d6fd5dd651 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:06.629168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433e9241-067e-46de-8db6-a9d6fd5dd651 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:06.635485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 360/719] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:06 2022] GET /identity/v3/services?type=network => generated 124 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:07.978803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd8b081-d165-43dc-ad48-5554088179c1 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:07.979528 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd8b081-d165-43dc-ad48-5554088179c1 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:07.979528 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd8b081-d165-43dc-ad48-5554088179c1 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:07.980991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 360/720] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:07.986382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714eee63-283e-4b37-b1ff-dbcc001ac4da None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:07.986877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714eee63-283e-4b37-b1ff-dbcc001ac4da None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:07.987039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714eee63-283e-4b37-b1ff-dbcc001ac4da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:08.039733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-714eee63-283e-4b37-b1ff-dbcc001ac4da None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:08.041023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-714eee63-283e-4b37-b1ff-dbcc001ac4da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:08.129567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 361/721] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:46:08.166374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd8b081-d165-43dc-ad48-5554088179c1 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:08.251373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:08.294017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd8b081-d165-43dc-ad48-5554088179c1 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:08.336258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab622ab-fdeb-422b-8065-17908fec06b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:08.337088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab622ab-fdeb-422b-8065-17908fec06b8 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:08.337369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab622ab-fdeb-422b-8065-17908fec06b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:08.337640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab622ab-fdeb-422b-8065-17908fec06b8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:08.348145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab622ab-fdeb-422b-8065-17908fec06b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-0ab622ab-fdeb-422b-8065-17908fec06b8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:08.364175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:08.365226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:46:08.366107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:08.368445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 361/722] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:46:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:08.371174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714eee63-283e-4b37-b1ff-dbcc001ac4da None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:08.529229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:08.532341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714eee63-283e-4b37-b1ff-dbcc001ac4da None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:08.578384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85444a24-43c0-4bd1-b836-f612dbcbd964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:08.579072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85444a24-43c0-4bd1-b836-f612dbcbd964 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:08.579360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85444a24-43c0-4bd1-b836-f612dbcbd964 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:08.579639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85444a24-43c0-4bd1-b836-f612dbcbd964 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:08.580398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85444a24-43c0-4bd1-b836-f612dbcbd964 None admin] RBAC: Adding query filter params (name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:08.580707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85444a24-43c0-4bd1-b836-f612dbcbd964 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:08.582135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85444a24-43c0-4bd1-b836-f612dbcbd964 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:08.596543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 362/723] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:46:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:08.602632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab622ab-fdeb-422b-8065-17908fec06b8 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:08.747492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab622ab-fdeb-422b-8065-17908fec06b8 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:08.786618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:08.800472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f536a355-c025-47f9-af40-449bdc9ace52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:08.801558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536a355-c025-47f9-af40-449bdc9ace52 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:08.815455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536a355-c025-47f9-af40-449bdc9ace52 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:08.815455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f536a355-c025-47f9-af40-449bdc9ace52 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:08.818717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f536a355-c025-47f9-af40-449bdc9ace52 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:08.820477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f536a355-c025-47f9-af40-449bdc9ace52 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-f536a355-c025-47f9-af40-449bdc9ace52 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:08.839651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:08.840634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:08.841659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:08.841659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:08.841659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:08.841659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:08.844072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 362/724] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:08.853976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85444a24-43c0-4bd1-b836-f612dbcbd964 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:08.917800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51acce1-cd72-4b4a-9c41-14fd6f572164 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:08.918134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51acce1-cd72-4b4a-9c41-14fd6f572164 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:08.918362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f51acce1-cd72-4b4a-9c41-14fd6f572164 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:08.920195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 363/725] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:08.925947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36360b1-0261-4fff-9040-dcedac392ea5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:08.926243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36360b1-0261-4fff-9040-dcedac392ea5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:08.926526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36360b1-0261-4fff-9040-dcedac392ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:08.945547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a36360b1-0261-4fff-9040-dcedac392ea5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:08.946738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a36360b1-0261-4fff-9040-dcedac392ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:09.046430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 364/726] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:46:09.055518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049d8104-7a66-47cf-8541-03ba118e845a None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.055814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049d8104-7a66-47cf-8541-03ba118e845a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.063131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049d8104-7a66-47cf-8541-03ba118e845a None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.063131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 365/727] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:09.064235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068d1866-4538-4038-bb24-1974beab198c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.064661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068d1866-4538-4038-bb24-1974beab198c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.065113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068d1866-4538-4038-bb24-1974beab198c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.070182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85444a24-43c0-4bd1-b836-f612dbcbd964 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:09.091705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-068d1866-4538-4038-bb24-1974beab198c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:09.092995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-068d1866-4538-4038-bb24-1974beab198c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:09.112687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66609c1d-78ce-4e0f-b884-31150644d13c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:09.113435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66609c1d-78ce-4e0f-b884-31150644d13c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.113724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66609c1d-78ce-4e0f-b884-31150644d13c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.114003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66609c1d-78ce-4e0f-b884-31150644d13c None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.114727 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66609c1d-78ce-4e0f-b884-31150644d13c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:09.115037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66609c1d-78ce-4e0f-b884-31150644d13c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:09.116458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66609c1d-78ce-4e0f-b884-31150644d13c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:09.130231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 363/728] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 277 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:09.131148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66609c1d-78ce-4e0f-b884-31150644d13c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:09.157340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:09.170620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 366/729] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jun 10 12:46:09.172100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ed24d-e3a3-4f82-a728-c3c8c6a4d750 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.172563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ed24d-e3a3-4f82-a728-c3c8c6a4d750 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.173041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1ed24d-e3a3-4f82-a728-c3c8c6a4d750 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.174735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 367/730] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:09.175651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1ed24d-e3a3-4f82-a728-c3c8c6a4d750 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:09.220795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:09.240986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66609c1d-78ce-4e0f-b884-31150644d13c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:09.277168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0d475e-07a9-42ad-9ef9-0c56c08c676a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:09.278068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0d475e-07a9-42ad-9ef9-0c56c08c676a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.278466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0d475e-07a9-42ad-9ef9-0c56c08c676a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.278748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0d475e-07a9-42ad-9ef9-0c56c08c676a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.282129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1ed24d-e3a3-4f82-a728-c3c8c6a4d750 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:09.289129 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0d475e-07a9-42ad-9ef9-0c56c08c676a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-bd0d475e-07a9-42ad-9ef9-0c56c08c676a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:09.297471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:09.298390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:09.299221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 364/731] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:09.299937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba29936-67dd-4f0a-8385-be8307ace7ca None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.300497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba29936-67dd-4f0a-8385-be8307ace7ca None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.300497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba29936-67dd-4f0a-8385-be8307ace7ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.318170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5ba29936-67dd-4f0a-8385-be8307ace7ca None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:09.322445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5ba29936-67dd-4f0a-8385-be8307ace7ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:09.326891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad62d2f-9d8a-4777-a485-d58528df6312 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:09.327777 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad62d2f-9d8a-4777-a485-d58528df6312 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.328064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad62d2f-9d8a-4777-a485-d58528df6312 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.328343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad62d2f-9d8a-4777-a485-d58528df6312 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.342249 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad62d2f-9d8a-4777-a485-d58528df6312 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:09.343743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad62d2f-9d8a-4777-a485-d58528df6312 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-4ad62d2f-9d8a-4777-a485-d58528df6312 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:09.360226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:09.361294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:09.362251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 368/732] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:46:09 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:09.362843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad62d2f-9d8a-4777-a485-d58528df6312 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:09.403764 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 365/733] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:46:09.404620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba29936-67dd-4f0a-8385-be8307ace7ca None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:09.428192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:09.461005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:09.464415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad62d2f-9d8a-4777-a485-d58528df6312 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:09.497107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545760a1-ce53-46f7-b8ab-fdb15d846a5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:09.498080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545760a1-ce53-46f7-b8ab-fdb15d846a5d None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.498423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545760a1-ce53-46f7-b8ab-fdb15d846a5d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.498998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545760a1-ce53-46f7-b8ab-fdb15d846a5d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.509037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545760a1-ce53-46f7-b8ab-fdb15d846a5d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:09.510399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba29936-67dd-4f0a-8385-be8307ace7ca None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-545760a1-ce53-46f7-b8ab-fdb15d846a5d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:09.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:09.517494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:09.518371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 369/734] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:09.519074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545760a1-ce53-46f7-b8ab-fdb15d846a5d None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:09.548096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13a02edc-d8ff-4070-8225-713a94164ec6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:09.548690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a02edc-d8ff-4070-8225-713a94164ec6 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.549008 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a02edc-d8ff-4070-8225-713a94164ec6 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.549287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13a02edc-d8ff-4070-8225-713a94164ec6 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.550024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a02edc-d8ff-4070-8225-713a94164ec6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:09.550328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a02edc-d8ff-4070-8225-713a94164ec6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:09.551707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13a02edc-d8ff-4070-8225-713a94164ec6 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:09.560029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 366/735] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:09.561490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501f7abd-da9e-4498-bcf0-e4785864b831 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.561780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501f7abd-da9e-4498-bcf0-e4785864b831 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.562060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501f7abd-da9e-4498-bcf0-e4785864b831 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.563266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 367/736] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:09.563936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501f7abd-da9e-4498-bcf0-e4785864b831 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:09.623797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545760a1-ce53-46f7-b8ab-fdb15d846a5d None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:09.638703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:09.656340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64179f8f-ddff-4ddf-a6ea-7fb40a78038e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:09.657020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64179f8f-ddff-4ddf-a6ea-7fb40a78038e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.657300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64179f8f-ddff-4ddf-a6ea-7fb40a78038e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.657574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64179f8f-ddff-4ddf-a6ea-7fb40a78038e None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.658397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64179f8f-ddff-4ddf-a6ea-7fb40a78038e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:09.658703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64179f8f-ddff-4ddf-a6ea-7fb40a78038e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:09.660121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64179f8f-ddff-4ddf-a6ea-7fb40a78038e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:09.664601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501f7abd-da9e-4498-bcf0-e4785864b831 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:09.677144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 370/737] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Jun 10 12:46:09 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:09.677923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64179f8f-ddff-4ddf-a6ea-7fb40a78038e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:09.695132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:09.702953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853aed05-bf24-4cdf-977c-0298fdb67998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:09.703744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853aed05-bf24-4cdf-977c-0298fdb67998 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.704032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853aed05-bf24-4cdf-977c-0298fdb67998 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.704306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853aed05-bf24-4cdf-977c-0298fdb67998 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.705040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853aed05-bf24-4cdf-977c-0298fdb67998 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:09.706416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853aed05-bf24-4cdf-977c-0298fdb67998 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:09.772972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64179f8f-ddff-4ddf-a6ea-7fb40a78038e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:09.800218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 368/738] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:46:09 2022] POST /identity/v3/services => generated 251 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:46:09.801216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853aed05-bf24-4cdf-977c-0298fdb67998 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:09.825390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c3bbe7-5d66-4bab-bd7b-9b9262028b6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:09.826072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c3bbe7-5d66-4bab-bd7b-9b9262028b6a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.826353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c3bbe7-5d66-4bab-bd7b-9b9262028b6a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.826628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c3bbe7-5d66-4bab-bd7b-9b9262028b6a None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.827422 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c3bbe7-5d66-4bab-bd7b-9b9262028b6a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:09.827698 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c3bbe7-5d66-4bab-bd7b-9b9262028b6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:09.829154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c3bbe7-5d66-4bab-bd7b-9b9262028b6a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:09.844394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 371/739] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:09.844394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590d1932-6e56-43dd-9066-b76c5e7501e3 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.844394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590d1932-6e56-43dd-9066-b76c5e7501e3 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.844394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590d1932-6e56-43dd-9066-b76c5e7501e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.859051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-590d1932-6e56-43dd-9066-b76c5e7501e3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:09.860277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-590d1932-6e56-43dd-9066-b76c5e7501e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:09.930739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853aed05-bf24-4cdf-977c-0298fdb67998 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:09.944794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 372/740] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:09 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jun 10 12:46:09.944794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590d1932-6e56-43dd-9066-b76c5e7501e3 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:09.948723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:09.981271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792832ed-678b-439e-8e2b-60cd339c967c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:09.982064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792832ed-678b-439e-8e2b-60cd339c967c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:09.982346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792832ed-678b-439e-8e2b-60cd339c967c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:09.982618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-792832ed-678b-439e-8e2b-60cd339c967c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:09.984541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:09.995226 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792832ed-678b-439e-8e2b-60cd339c967c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:09.996681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-792832ed-678b-439e-8e2b-60cd339c967c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-792832ed-678b-439e-8e2b-60cd339c967c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:10.015481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:10.016397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:10.017673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:10.017673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:10.017673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:10.017673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:10.017673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 369/741] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:10.017673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792832ed-678b-439e-8e2b-60cd339c967c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:10.065939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590d1932-6e56-43dd-9066-b76c5e7501e3 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:10.101914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adfea64-d160-4083-81b9-6610d2b9f48e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:10.102567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adfea64-d160-4083-81b9-6610d2b9f48e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:10.102850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adfea64-d160-4083-81b9-6610d2b9f48e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:10.103127 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6adfea64-d160-4083-81b9-6610d2b9f48e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:10.125239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-792832ed-678b-439e-8e2b-60cd339c967c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:10.166909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df33a291-654e-4171-a386-7ffedaf74276 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:10.167703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df33a291-654e-4171-a386-7ffedaf74276 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:10.167983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df33a291-654e-4171-a386-7ffedaf74276 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:10.168255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df33a291-654e-4171-a386-7ffedaf74276 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:10.176137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adfea64-d160-4083-81b9-6610d2b9f48e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:10.177787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6adfea64-d160-4083-81b9-6610d2b9f48e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:10.179742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df33a291-654e-4171-a386-7ffedaf74276 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:10.180426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:10.181928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df33a291-654e-4171-a386-7ffedaf74276 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-df33a291-654e-4171-a386-7ffedaf74276 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:10.199807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:10.200740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:10.201761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:10.201761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:10.201761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:10.201761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:10.201761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 370/742] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:10.202102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df33a291-654e-4171-a386-7ffedaf74276 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:10.249419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:10.286208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 373/743] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:09 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jun 10 12:46:10.287048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adfea64-d160-4083-81b9-6610d2b9f48e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:10.317796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df33a291-654e-4171-a386-7ffedaf74276 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:10.370889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153c036a-36c3-403b-a8c1-d2ededdfbb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:10.371873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153c036a-36c3-403b-a8c1-d2ededdfbb3a None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:10.372428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153c036a-36c3-403b-a8c1-d2ededdfbb3a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:10.372714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153c036a-36c3-403b-a8c1-d2ededdfbb3a None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:10.373490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153c036a-36c3-403b-a8c1-d2ededdfbb3a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:10.375543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-153c036a-36c3-403b-a8c1-d2ededdfbb3a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:10.392241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6adfea64-d160-4083-81b9-6610d2b9f48e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:10.394504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:10.425569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce670d5-04f8-4eb1-9c45-fb8b37f9143a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:10.426219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce670d5-04f8-4eb1-9c45-fb8b37f9143a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:10.426507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce670d5-04f8-4eb1-9c45-fb8b37f9143a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:10.426785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ce670d5-04f8-4eb1-9c45-fb8b37f9143a None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:10.427502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce670d5-04f8-4eb1-9c45-fb8b37f9143a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:10.427810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce670d5-04f8-4eb1-9c45-fb8b37f9143a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:10.429238 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ce670d5-04f8-4eb1-9c45-fb8b37f9143a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:10.440130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 374/744] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:10.440928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce670d5-04f8-4eb1-9c45-fb8b37f9143a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:10.472796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 371/745] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:46:10 2022] POST /identity/v3/services => generated 240 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:46:10.472796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153c036a-36c3-403b-a8c1-d2ededdfbb3a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:10.568358 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ce670d5-04f8-4eb1-9c45-fb8b37f9143a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:10.610427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:10.620758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153c036a-36c3-403b-a8c1-d2ededdfbb3a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:10.621499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6968ea4d-279c-4b95-b53b-2fa6eca02037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:10.622374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6968ea4d-279c-4b95-b53b-2fa6eca02037 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:10.623010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6968ea4d-279c-4b95-b53b-2fa6eca02037 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:10.625182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6968ea4d-279c-4b95-b53b-2fa6eca02037 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:10.625550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6968ea4d-279c-4b95-b53b-2fa6eca02037 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:10.626074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6968ea4d-279c-4b95-b53b-2fa6eca02037 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:10.627724 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6968ea4d-279c-4b95-b53b-2fa6eca02037 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:10.645092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 375/746] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:10.646345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6968ea4d-279c-4b95-b53b-2fa6eca02037 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:10.665456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2ae77f-7a73-4eb6-8482-f82dd96f6db4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:10.666250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2ae77f-7a73-4eb6-8482-f82dd96f6db4 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:10.666538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2ae77f-7a73-4eb6-8482-f82dd96f6db4 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:10.666814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2ae77f-7a73-4eb6-8482-f82dd96f6db4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:10.679577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:10.681278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2ae77f-7a73-4eb6-8482-f82dd96f6db4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:10.681863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b2ae77f-7a73-4eb6-8482-f82dd96f6db4 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-1b2ae77f-7a73-4eb6-8482-f82dd96f6db4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:10.708810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:10.709819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:10.711278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:10.711278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:10.711278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:10.711278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:10.711515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 372/747] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:10.712309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2ae77f-7a73-4eb6-8482-f82dd96f6db4 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:10.777704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6968ea4d-279c-4b95-b53b-2fa6eca02037 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:10.814291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89dc092-69c5-4b64-950b-7ff0fd22bdec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:10.815467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89dc092-69c5-4b64-950b-7ff0fd22bdec None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:10.815467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89dc092-69c5-4b64-950b-7ff0fd22bdec None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:10.815887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89dc092-69c5-4b64-950b-7ff0fd22bdec None admin] PATH_INFO: `/v3/users/nova` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:10.821054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2ae77f-7a73-4eb6-8482-f82dd96f6db4 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:10.824550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89dc092-69c5-4b64-950b-7ff0fd22bdec None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:10.825697 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89dc092-69c5-4b64-950b-7ff0fd22bdec None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-e89dc092-69c5-4b64-950b-7ff0fd22bdec None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:10.841700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:10.842671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:10.843603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:10.847606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 376/748] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Jun 10 12:46:10 2022] GET /identity/v3/users/nova => generated 82 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:10.848468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89dc092-69c5-4b64-950b-7ff0fd22bdec None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:10.861964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bb1ab6-3b25-4be0-a1b0-dd0a1b766225 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:10.863129 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb1ab6-3b25-4be0-a1b0-dd0a1b766225 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:10.863612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb1ab6-3b25-4be0-a1b0-dd0a1b766225 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:10.864081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17bb1ab6-3b25-4be0-a1b0-dd0a1b766225 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:10.871736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bb1ab6-3b25-4be0-a1b0-dd0a1b766225 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:10.873446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17bb1ab6-3b25-4be0-a1b0-dd0a1b766225 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-17bb1ab6-3b25-4be0-a1b0-dd0a1b766225 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:10.888195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:10.889152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:46:10.890099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:10.892801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 373/749] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:10 2022] GET /identity/v3/users/swift => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:10.893933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bb1ab6-3b25-4be0-a1b0-dd0a1b766225 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:10.903693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:10.946034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:10.958639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89dc092-69c5-4b64-950b-7ff0fd22bdec None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:10.997449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b1d25-2321-4a20-98d9-d6b54b34cc6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:10.998461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b1d25-2321-4a20-98d9-d6b54b34cc6e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:10.998902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b1d25-2321-4a20-98d9-d6b54b34cc6e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:10.999330 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7b1d25-2321-4a20-98d9-d6b54b34cc6e None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.000210 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b1d25-2321-4a20-98d9-d6b54b34cc6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:11.000662 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b1d25-2321-4a20-98d9-d6b54b34cc6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.001743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17bb1ab6-3b25-4be0-a1b0-dd0a1b766225 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.002399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7b1d25-2321-4a20-98d9-d6b54b34cc6e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.015673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 377/750] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:11.029787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b1d25-2321-4a20-98d9-d6b54b34cc6e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.040908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061920b4-68dd-4a5f-b4a8-a14b69b8d14a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.041582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061920b4-68dd-4a5f-b4a8-a14b69b8d14a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.041863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061920b4-68dd-4a5f-b4a8-a14b69b8d14a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.042138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061920b4-68dd-4a5f-b4a8-a14b69b8d14a None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.042976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061920b4-68dd-4a5f-b4a8-a14b69b8d14a None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:11.043281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061920b4-68dd-4a5f-b4a8-a14b69b8d14a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.044794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061920b4-68dd-4a5f-b4a8-a14b69b8d14a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.063465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 374/751] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Jun 10 12:46:10 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:11.064340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061920b4-68dd-4a5f-b4a8-a14b69b8d14a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.129063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:11.133951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7b1d25-2321-4a20-98d9-d6b54b34cc6e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.168804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061920b4-68dd-4a5f-b4a8-a14b69b8d14a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.170820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415d25a7-6637-4355-8be6-05eebff35d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.171585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415d25a7-6637-4355-8be6-05eebff35d5a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.171878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415d25a7-6637-4355-8be6-05eebff35d5a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.172154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415d25a7-6637-4355-8be6-05eebff35d5a None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.173034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415d25a7-6637-4355-8be6-05eebff35d5a None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:11.173342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415d25a7-6637-4355-8be6-05eebff35d5a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.174786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-415d25a7-6637-4355-8be6-05eebff35d5a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.193848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 378/752] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:11.195410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415d25a7-6637-4355-8be6-05eebff35d5a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.198217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:11.203680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9041f413-46d8-4001-a848-fd65b363241a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.204472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9041f413-46d8-4001-a848-fd65b363241a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.204752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9041f413-46d8-4001-a848-fd65b363241a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.205055 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9041f413-46d8-4001-a848-fd65b363241a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.212015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9041f413-46d8-4001-a848-fd65b363241a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.213489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9041f413-46d8-4001-a848-fd65b363241a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-9041f413-46d8-4001-a848-fd65b363241a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:11.228595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:11.230721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 375/753] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:11.232797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9041f413-46d8-4001-a848-fd65b363241a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.306237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415d25a7-6637-4355-8be6-05eebff35d5a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.333108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9041f413-46d8-4001-a848-fd65b363241a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.345917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab8c96e-1bec-4196-9676-3f77919692d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.347016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab8c96e-1bec-4196-9676-3f77919692d4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.347508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab8c96e-1bec-4196-9676-3f77919692d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.347927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab8c96e-1bec-4196-9676-3f77919692d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.357469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:11.361694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab8c96e-1bec-4196-9676-3f77919692d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.363334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab8c96e-1bec-4196-9676-3f77919692d4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.368588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2cf3c2-b4cd-4345-95bb-56c5685e8797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.374340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2cf3c2-b4cd-4345-95bb-56c5685e8797 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.374340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2cf3c2-b4cd-4345-95bb-56c5685e8797 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.374340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e2cf3c2-b4cd-4345-95bb-56c5685e8797 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-0ab8c96e-1bec-4196-9676-3f77919692d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:11.383511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:11.384571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:11.385617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:11.385617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:11.385617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:11.385617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:11.391645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 379/754] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:11.392650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab8c96e-1bec-4196-9676-3f77919692d4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.440339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2cf3c2-b4cd-4345-95bb-56c5685e8797 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.441676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e2cf3c2-b4cd-4345-95bb-56c5685e8797 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.453888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:11.502413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab8c96e-1bec-4196-9676-3f77919692d4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.539393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214e8453-c081-4e3b-900b-217c315812b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.540513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214e8453-c081-4e3b-900b-217c315812b2 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.541075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214e8453-c081-4e3b-900b-217c315812b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.541500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214e8453-c081-4e3b-900b-217c315812b2 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.542429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214e8453-c081-4e3b-900b-217c315812b2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:11.542884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214e8453-c081-4e3b-900b-217c315812b2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.544270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 376/755] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:46:11.544725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214e8453-c081-4e3b-900b-217c315812b2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.545143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2cf3c2-b4cd-4345-95bb-56c5685e8797 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.559652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 380/756] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:11.560828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214e8453-c081-4e3b-900b-217c315812b2 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.575494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:11.652392 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e2cf3c2-b4cd-4345-95bb-56c5685e8797 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.671763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214e8453-c081-4e3b-900b-217c315812b2 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.688825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5b205a-77c3-46c5-ba34-b0c54a067bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.689085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:11.689776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5b205a-77c3-46c5-ba34-b0c54a067bf3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.689895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5b205a-77c3-46c5-ba34-b0c54a067bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.690103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5b205a-77c3-46c5-ba34-b0c54a067bf3 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.691359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5b205a-77c3-46c5-ba34-b0c54a067bf3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:11.691770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5b205a-77c3-46c5-ba34-b0c54a067bf3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.693351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5b205a-77c3-46c5-ba34-b0c54a067bf3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.710340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 377/757] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:11.710340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5b205a-77c3-46c5-ba34-b0c54a067bf3 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.716386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0f435f-5f76-431c-b3b8-eeab59303756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.717410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0f435f-5f76-431c-b3b8-eeab59303756 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.717840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0f435f-5f76-431c-b3b8-eeab59303756 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.718264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0f435f-5f76-431c-b3b8-eeab59303756 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.730870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0f435f-5f76-431c-b3b8-eeab59303756 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.732619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0f435f-5f76-431c-b3b8-eeab59303756 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-0f0f435f-5f76-431c-b3b8-eeab59303756 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:11.752209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:11.753346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:11.754458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:11.754458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:11.754458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:11.754458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:11.756455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 381/758] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:11.757559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0f435f-5f76-431c-b3b8-eeab59303756 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.811769 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:11.829052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5b205a-77c3-46c5-ba34-b0c54a067bf3 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.872746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd72966-cec0-4abb-8354-415f9a741246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.873807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd72966-cec0-4abb-8354-415f9a741246 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.874298 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd72966-cec0-4abb-8354-415f9a741246 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.874749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd72966-cec0-4abb-8354-415f9a741246 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.875147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0f435f-5f76-431c-b3b8-eeab59303756 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:11.875942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd72966-cec0-4abb-8354-415f9a741246 None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=eb2c6f06b300473ebafff4503bb24396) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:11.876428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd72966-cec0-4abb-8354-415f9a741246 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.879180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd72966-cec0-4abb-8354-415f9a741246 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.887265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 378/759] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/role_assignments?user.id=eb2c6f06b300473ebafff4503bb24396&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 259 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:11.888645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd72966-cec0-4abb-8354-415f9a741246 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:11.924649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3776a4-86b1-4900-bfcf-d2ce288a8373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:11.925821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3776a4-86b1-4900-bfcf-d2ce288a8373 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:11.926269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3776a4-86b1-4900-bfcf-d2ce288a8373 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:11.926706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3776a4-86b1-4900-bfcf-d2ce288a8373 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:11.935494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3776a4-86b1-4900-bfcf-d2ce288a8373 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:11.937417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3776a4-86b1-4900-bfcf-d2ce288a8373 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-4b3776a4-86b1-4900-bfcf-d2ce288a8373 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:11.953638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:11.954771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:11.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:11.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:11.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:11.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:11.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:11.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:11.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:11.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:11.958016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 382/760] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:11.975329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3776a4-86b1-4900-bfcf-d2ce288a8373 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:12.023074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd72966-cec0-4abb-8354-415f9a741246 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:12.023703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:12.061977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70004d96-e091-495b-9fcc-22ce625ff709 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:12.062814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70004d96-e091-495b-9fcc-22ce625ff709 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.063285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70004d96-e091-495b-9fcc-22ce625ff709 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.063631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70004d96-e091-495b-9fcc-22ce625ff709 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.064439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70004d96-e091-495b-9fcc-22ce625ff709 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:12.064865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70004d96-e091-495b-9fcc-22ce625ff709 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:12.066360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70004d96-e091-495b-9fcc-22ce625ff709 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:12.079925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 379/761] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:12.082184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d157a714-2691-499f-905f-556ba9a9109d None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.082541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d157a714-2691-499f-905f-556ba9a9109d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.083042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d157a714-2691-499f-905f-556ba9a9109d None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.084680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 380/762] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:12.087625 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d157a714-2691-499f-905f-556ba9a9109d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:12.099258 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:12.107402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3776a4-86b1-4900-bfcf-d2ce288a8373 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:12.150868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835e7b4b-35ba-43e2-ab5b-da8710d037fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:12.152063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835e7b4b-35ba-43e2-ab5b-da8710d037fb None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.152345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835e7b4b-35ba-43e2-ab5b-da8710d037fb None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.152621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835e7b4b-35ba-43e2-ab5b-da8710d037fb None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.153693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835e7b4b-35ba-43e2-ab5b-da8710d037fb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:12.154395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835e7b4b-35ba-43e2-ab5b-da8710d037fb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:12.156102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835e7b4b-35ba-43e2-ab5b-da8710d037fb None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:12.173990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 383/763] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:12.175643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f81cb9c-810a-44d3-8c0e-c4dbb4087d7c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.175981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f81cb9c-810a-44d3-8c0e-c4dbb4087d7c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.176458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f81cb9c-810a-44d3-8c0e-c4dbb4087d7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.196092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-7f81cb9c-810a-44d3-8c0e-c4dbb4087d7c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:12.198087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7f81cb9c-810a-44d3-8c0e-c4dbb4087d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:12.201526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d157a714-2691-499f-905f-556ba9a9109d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:12.240871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806dff32-3605-4e81-9105-539a0723339e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:12.242283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806dff32-3605-4e81-9105-539a0723339e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.242703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806dff32-3605-4e81-9105-539a0723339e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.242981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806dff32-3605-4e81-9105-539a0723339e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.251197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806dff32-3605-4e81-9105-539a0723339e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:12.252665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806dff32-3605-4e81-9105-539a0723339e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-806dff32-3605-4e81-9105-539a0723339e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:12.269700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:12.270640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:12.271515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:12.271515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:12.271515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:12.271515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:12.271515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:12.271515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:12.271515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:12.271515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:12.271515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 381/764] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:12.271993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806dff32-3605-4e81-9105-539a0723339e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:12.295231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 384/765] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:12.296152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f81cb9c-810a-44d3-8c0e-c4dbb4087d7c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:12.315357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:12.369949 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:12.406831 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806dff32-3605-4e81-9105-539a0723339e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:12.437254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f81cb9c-810a-44d3-8c0e-c4dbb4087d7c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:12.447735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b162aa18-7c0e-4401-a38c-44ddb1063cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:12.448711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b162aa18-7c0e-4401-a38c-44ddb1063cc3 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.449093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b162aa18-7c0e-4401-a38c-44ddb1063cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.449441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b162aa18-7c0e-4401-a38c-44ddb1063cc3 None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/e9a0496199134369a0ef740c099eed59/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.452979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b162aa18-7c0e-4401-a38c-44ddb1063cc3 None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=e9a0496199134369a0ef740c099eed59, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:12.454773 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b162aa18-7c0e-4401-a38c-44ddb1063cc3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:12.477460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b409b26-9fe4-4c79-bd81-fda916bddb55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:12.478121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b409b26-9fe4-4c79-bd81-fda916bddb55 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.478405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b409b26-9fe4-4c79-bd81-fda916bddb55 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.478828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b409b26-9fe4-4c79-bd81-fda916bddb55 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.479540 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b409b26-9fe4-4c79-bd81-fda916bddb55 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:12.479850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b409b26-9fe4-4c79-bd81-fda916bddb55 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:12.481329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b409b26-9fe4-4c79-bd81-fda916bddb55 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:12.493824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 385/766] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:12.494620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b409b26-9fe4-4c79-bd81-fda916bddb55 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:12.534954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 382/767] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:46:12 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/e9a0496199134369a0ef740c099eed59/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:46:12.537008 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b162aa18-7c0e-4401-a38c-44ddb1063cc3 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:12.610536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:12.667130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b409b26-9fe4-4c79-bd81-fda916bddb55 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:12.675302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b162aa18-7c0e-4401-a38c-44ddb1063cc3 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:12.708864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:12.719456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f6e22-2836-481f-a7ab-832efbc1fd17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:12.720141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f6e22-2836-481f-a7ab-832efbc1fd17 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.720429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f6e22-2836-481f-a7ab-832efbc1fd17 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.720711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35f6e22-2836-481f-a7ab-832efbc1fd17 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.721678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35f6e22-2836-481f-a7ab-832efbc1fd17 None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=bb18cd41c9a14d358aac14ed62ef9e02) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:12.721994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35f6e22-2836-481f-a7ab-832efbc1fd17 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:12.724347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35f6e22-2836-481f-a7ab-832efbc1fd17 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:12.731536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 383/768] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:12 2022] GET /identity/v3/role_assignments?user.id=bb18cd41c9a14d358aac14ed62ef9e02&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=766799d1b6c049168590571f1a959b3b => generated 259 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:12.763859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a5a893-be56-4689-aa2c-b06c4f879af7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:12.764790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a5a893-be56-4689-aa2c-b06c4f879af7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.765090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a5a893-be56-4689-aa2c-b06c4f879af7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.765370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a5a893-be56-4689-aa2c-b06c4f879af7 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.766060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a5a893-be56-4689-aa2c-b06c4f879af7 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:12.767592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a5a893-be56-4689-aa2c-b06c4f879af7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-a7a5a893-be56-4689-aa2c-b06c4f879af7 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:12.783988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:12.785035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:12.785879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 386/769] 10.222.0.25 () {56 vars in 1129 bytes} [Fri Jun 10 12:46:12 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:12.792236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f6e22-2836-481f-a7ab-832efbc1fd17 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:12.833980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54079000-4212-4c8f-82c1-dad1fc4abf5d None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.834300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54079000-4212-4c8f-82c1-dad1fc4abf5d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.834542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54079000-4212-4c8f-82c1-dad1fc4abf5d None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.836477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 387/770] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:12.842673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e99e0ce-5488-4861-aa90-4fcdd7da6f20 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.843114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e99e0ce-5488-4861-aa90-4fcdd7da6f20 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.843364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e99e0ce-5488-4861-aa90-4fcdd7da6f20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.868293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-0e99e0ce-5488-4861-aa90-4fcdd7da6f20 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:12.869546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0e99e0ce-5488-4861-aa90-4fcdd7da6f20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:12.924670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35f6e22-2836-481f-a7ab-832efbc1fd17 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:12.976958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 388/771] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:12.977250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c396875-232e-42bf-9cc9-f026b7d238a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:12.977670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c396875-232e-42bf-9cc9-f026b7d238a4 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:12.977956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c396875-232e-42bf-9cc9-f026b7d238a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:12.978237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c396875-232e-42bf-9cc9-f026b7d238a4 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:12.979009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c396875-232e-42bf-9cc9-f026b7d238a4 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:12.979320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c396875-232e-42bf-9cc9-f026b7d238a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:12.980703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c396875-232e-42bf-9cc9-f026b7d238a4 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:12.990101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 384/772] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Jun 10 12:46:12 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:12.994908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e99e0ce-5488-4861-aa90-4fcdd7da6f20 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:12.995554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c396875-232e-42bf-9cc9-f026b7d238a4 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:13.095022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:13.107267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:13.167261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e99e0ce-5488-4861-aa90-4fcdd7da6f20 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:13.179598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c396875-232e-42bf-9cc9-f026b7d238a4 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:13.229048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d704ba-7839-4eca-9388-dfc6aa01310c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:13.229762 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d704ba-7839-4eca-9388-dfc6aa01310c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:13.230040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d704ba-7839-4eca-9388-dfc6aa01310c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:13.230304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d704ba-7839-4eca-9388-dfc6aa01310c None admin] PATH_INFO: `/v3/services/network` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:13.230960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d704ba-7839-4eca-9388-dfc6aa01310c None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:13.232336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d704ba-7839-4eca-9388-dfc6aa01310c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:13.248018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fac2824-1a7c-43f4-a5e8-a5904d6f0eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:13.248679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fac2824-1a7c-43f4-a5e8-a5904d6f0eed None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:13.248981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fac2824-1a7c-43f4-a5e8-a5904d6f0eed None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:13.249263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fac2824-1a7c-43f4-a5e8-a5904d6f0eed None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:13.250152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fac2824-1a7c-43f4-a5e8-a5904d6f0eed None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:13.250461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fac2824-1a7c-43f4-a5e8-a5904d6f0eed None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:13.251835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fac2824-1a7c-43f4-a5e8-a5904d6f0eed None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-e7d704ba-7839-4eca-9388-dfc6aa01310c None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:13.253595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:13.254545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jun 10 12:46:13.255354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:13.261086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 389/773] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:12 2022] GET /identity/v3/services/network => generated 88 bytes in 262 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:13.266956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 385/774] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Jun 10 12:46:12 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:13.268030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d704ba-7839-4eca-9388-dfc6aa01310c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:13.272351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fac2824-1a7c-43f4-a5e8-a5904d6f0eed None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:13.441277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fac2824-1a7c-43f4-a5e8-a5904d6f0eed None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:13.444922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d704ba-7839-4eca-9388-dfc6aa01310c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:13.475773 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:13.499461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11286ca-b2ff-49ff-afea-c1734230c7e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:13.500051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11286ca-b2ff-49ff-afea-c1734230c7e9 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:13.500335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11286ca-b2ff-49ff-afea-c1734230c7e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:13.500607 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11286ca-b2ff-49ff-afea-c1734230c7e9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:13.501419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11286ca-b2ff-49ff-afea-c1734230c7e9 None admin] RBAC: Adding query filter params (interface=public, service_id=b88443ea297f4a358368bb3d11ff7df4, region_id=RegionOne) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:13.501730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11286ca-b2ff-49ff-afea-c1734230c7e9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:13.503107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11286ca-b2ff-49ff-afea-c1734230c7e9 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:13.505113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60455725-5a84-4374-871d-8a15b581fd8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:13.505769 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60455725-5a84-4374-871d-8a15b581fd8c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:13.506140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60455725-5a84-4374-871d-8a15b581fd8c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:13.506414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60455725-5a84-4374-871d-8a15b581fd8c None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:13.507641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60455725-5a84-4374-871d-8a15b581fd8c None admin] RBAC: Adding query filter params (name=network) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:13.507945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60455725-5a84-4374-871d-8a15b581fd8c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:13.509327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60455725-5a84-4374-871d-8a15b581fd8c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:13.511704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:13.520954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 386/776] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Jun 10 12:46:13 2022] GET /identity/v3/endpoints?service_id=b88443ea297f4a358368bb3d11ff7df4&interface=public®ion_id=RegionOne => generated 194 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:13.524397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 390/776] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:13 2022] GET /identity/v3/services?name=network => generated 124 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:13.527703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60455725-5a84-4374-871d-8a15b581fd8c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:13.528965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11286ca-b2ff-49ff-afea-c1734230c7e9 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:13.721499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11286ca-b2ff-49ff-afea-c1734230c7e9 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:13.727129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60455725-5a84-4374-871d-8a15b581fd8c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:13.787609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a8134f-8b64-4f3c-9090-85a09ad26ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:13.788715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a8134f-8b64-4f3c-9090-85a09ad26ded None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:13.789020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a8134f-8b64-4f3c-9090-85a09ad26ded None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:13.789299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a8134f-8b64-4f3c-9090-85a09ad26ded None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:13.789981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a8134f-8b64-4f3c-9090-85a09ad26ded None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:13.791357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a8134f-8b64-4f3c-9090-85a09ad26ded None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:13.795386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99489ae-48b3-406e-aec3-5a94377a1134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:13.796038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99489ae-48b3-406e-aec3-5a94377a1134 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:13.796319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99489ae-48b3-406e-aec3-5a94377a1134 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:13.796593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99489ae-48b3-406e-aec3-5a94377a1134 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:13.797318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99489ae-48b3-406e-aec3-5a94377a1134 None admin] RBAC: Adding query filter params (type=network) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:13.797620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99489ae-48b3-406e-aec3-5a94377a1134 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:13.798968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99489ae-48b3-406e-aec3-5a94377a1134 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:13.805521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 387/777] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Jun 10 12:46:13 2022] GET /identity/v3/services => generated 772 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:13.811359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 391/778] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:13 2022] GET /identity/v3/services?type=network => generated 350 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:13.817033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a8134f-8b64-4f3c-9090-85a09ad26ded None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:13.894479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:13.964176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a8134f-8b64-4f3c-9090-85a09ad26ded None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:14.018449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-568a3a06-93c2-46dd-9f75-e55dc84f2463 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:14.019155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568a3a06-93c2-46dd-9f75-e55dc84f2463 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:14.019436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568a3a06-93c2-46dd-9f75-e55dc84f2463 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:14.019714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-568a3a06-93c2-46dd-9f75-e55dc84f2463 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:14.020539 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568a3a06-93c2-46dd-9f75-e55dc84f2463 None admin] RBAC: Adding query filter params (interface=public, service_id=c3cdfc70c62c483ca4d768222d0571e2, region_id=RegionOne) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:14.020879 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568a3a06-93c2-46dd-9f75-e55dc84f2463 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:14.022874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-568a3a06-93c2-46dd-9f75-e55dc84f2463 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:14.029703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 388/779] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Jun 10 12:46:13 2022] GET /identity/v3/endpoints?service_id=c3cdfc70c62c483ca4d768222d0571e2&interface=public®ion_id=RegionOne => generated 194 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:14.035333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99489ae-48b3-406e-aec3-5a94377a1134 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:14.118238 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:14.159348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99489ae-48b3-406e-aec3-5a94377a1134 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:14.194828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e92811a-1269-40fd-8d16-a2466280090f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:14.195585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e92811a-1269-40fd-8d16-a2466280090f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:14.195860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e92811a-1269-40fd-8d16-a2466280090f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:14.196148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e92811a-1269-40fd-8d16-a2466280090f None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:14.196876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e92811a-1269-40fd-8d16-a2466280090f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:14.198337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e92811a-1269-40fd-8d16-a2466280090f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:14.204874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 392/780] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Jun 10 12:46:14 2022] GET /identity/v3/services => generated 772 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:14.267182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a86515-3113-4b28-95ec-f573d336467b None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:14.267719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a86515-3113-4b28-95ec-f573d336467b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:14.267719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a86515-3113-4b28-95ec-f573d336467b None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:14.269431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 389/781] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:14.275451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462931dc-e523-44a0-9d6e-087a99726f93 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:14.275733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462931dc-e523-44a0-9d6e-087a99726f93 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:14.276282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462931dc-e523-44a0-9d6e-087a99726f93 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:14.302175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-462931dc-e523-44a0-9d6e-087a99726f93 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:14.303433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-462931dc-e523-44a0-9d6e-087a99726f93 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:14.431196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 393/782] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:14.454757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a86515-3113-4b28-95ec-f573d336467b None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:14.569995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a86515-3113-4b28-95ec-f573d336467b None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:14.592880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:14.614929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19775563-12fe-4ef1-992d-90b85aa91c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:14.616097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19775563-12fe-4ef1-992d-90b85aa91c06 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:14.616602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19775563-12fe-4ef1-992d-90b85aa91c06 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:14.617121 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19775563-12fe-4ef1-992d-90b85aa91c06 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:14.628107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19775563-12fe-4ef1-992d-90b85aa91c06 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-19775563-12fe-4ef1-992d-90b85aa91c06 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:14.636407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:14.637374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:46:14.638197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:14.641024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 390/783] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:46:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:14.648823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462931dc-e523-44a0-9d6e-087a99726f93 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:14.726615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:14.800125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462931dc-e523-44a0-9d6e-087a99726f93 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:14.857477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d0484e-b955-4553-a6b5-6f9edbd0f775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:14.858529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d0484e-b955-4553-a6b5-6f9edbd0f775 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:14.859037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d0484e-b955-4553-a6b5-6f9edbd0f775 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:14.859516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d0484e-b955-4553-a6b5-6f9edbd0f775 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:14.860508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d0484e-b955-4553-a6b5-6f9edbd0f775 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:14.861032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d0484e-b955-4553-a6b5-6f9edbd0f775 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:14.862600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d0484e-b955-4553-a6b5-6f9edbd0f775 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:14.874789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 394/784] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:46:14 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:14.880346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19775563-12fe-4ef1-992d-90b85aa91c06 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:15.011296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19775563-12fe-4ef1-992d-90b85aa91c06 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:15.058400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43bfd07-21b4-4c5e-bc17-77e357c1b654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:15.059562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43bfd07-21b4-4c5e-bc17-77e357c1b654 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.060087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43bfd07-21b4-4c5e-bc17-77e357c1b654 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.060598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43bfd07-21b4-4c5e-bc17-77e357c1b654 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.078388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43bfd07-21b4-4c5e-bc17-77e357c1b654 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:15.080159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43bfd07-21b4-4c5e-bc17-77e357c1b654 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-e43bfd07-21b4-4c5e-bc17-77e357c1b654 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:15.101803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:15.102745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:15.103647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:15.103647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:15.103647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:15.103647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:15.105940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 391/785] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:15.111194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d0484e-b955-4553-a6b5-6f9edbd0f775 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:15.252566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d0484e-b955-4553-a6b5-6f9edbd0f775 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:15.260576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf49ce1-d0a7-4d3f-b4d7-ede1fbb81670 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.261197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf49ce1-d0a7-4d3f-b4d7-ede1fbb81670 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.261720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf49ce1-d0a7-4d3f-b4d7-ede1fbb81670 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.263559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 392/786] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:15.273455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc4fba6-083c-4c0b-ab9d-aa752cea25df None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.274490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc4fba6-083c-4c0b-ab9d-aa752cea25df None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.274961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc4fba6-083c-4c0b-ab9d-aa752cea25df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.293063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:15.295644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bcc4fba6-083c-4c0b-ab9d-aa752cea25df None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:15.298976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bcc4fba6-083c-4c0b-ab9d-aa752cea25df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:15.313937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ac53b35-7600-452b-81cf-204ad05db819 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:15.315075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac53b35-7600-452b-81cf-204ad05db819 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.315555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac53b35-7600-452b-81cf-204ad05db819 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.316149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac53b35-7600-452b-81cf-204ad05db819 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.317265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac53b35-7600-452b-81cf-204ad05db819 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:15.317887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac53b35-7600-452b-81cf-204ad05db819 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:15.319509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ac53b35-7600-452b-81cf-204ad05db819 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:15.323109 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:15.337669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 395/787] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:15.337669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84213996-a0f6-46de-9ea1-f5323d22dd02 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.338226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84213996-a0f6-46de-9ea1-f5323d22dd02 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.338694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84213996-a0f6-46de-9ea1-f5323d22dd02 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.340565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 396/788] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:15.341507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84213996-a0f6-46de-9ea1-f5323d22dd02 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:15.407226 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 393/789] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:15.409430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e62b67-bd39-4fc1-95ef-2dc1a9cca2b2 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.409890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e62b67-bd39-4fc1-95ef-2dc1a9cca2b2 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.410310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e62b67-bd39-4fc1-95ef-2dc1a9cca2b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.445378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-14e62b67-bd39-4fc1-95ef-2dc1a9cca2b2 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:15.446562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-14e62b67-bd39-4fc1-95ef-2dc1a9cca2b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:15.475852 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84213996-a0f6-46de-9ea1-f5323d22dd02 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:15.513451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0095c4f6-89ba-4274-b6b5-30bdb4a9eb3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:15.514065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0095c4f6-89ba-4274-b6b5-30bdb4a9eb3b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.514354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0095c4f6-89ba-4274-b6b5-30bdb4a9eb3b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.514633 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0095c4f6-89ba-4274-b6b5-30bdb4a9eb3b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.524609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0095c4f6-89ba-4274-b6b5-30bdb4a9eb3b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:15.526363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0095c4f6-89ba-4274-b6b5-30bdb4a9eb3b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:15.536845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 394/790] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:15 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:15.537783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e62b67-bd39-4fc1-95ef-2dc1a9cca2b2 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-0095c4f6-89ba-4274-b6b5-30bdb4a9eb3b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:15.545195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:15.546177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:15.547861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 397/791] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:46:15 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:15.554343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0095c4f6-89ba-4274-b6b5-30bdb4a9eb3b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:15.612711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:15.650025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:15.666616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e62b67-bd39-4fc1-95ef-2dc1a9cca2b2 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:15.698125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0095c4f6-89ba-4274-b6b5-30bdb4a9eb3b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:15.707302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86d6c61-9aa4-4e3c-ab24-5b1c5bf69e92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:15.708102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86d6c61-9aa4-4e3c-ab24-5b1c5bf69e92 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.708379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86d6c61-9aa4-4e3c-ab24-5b1c5bf69e92 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.708720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86d6c61-9aa4-4e3c-ab24-5b1c5bf69e92 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.725628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c86d6c61-9aa4-4e3c-ab24-5b1c5bf69e92 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-c86d6c61-9aa4-4e3c-ab24-5b1c5bf69e92 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:15.736619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:15.737566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:15.738389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:15.739007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 395/792] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:15.739335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86d6c61-9aa4-4e3c-ab24-5b1c5bf69e92 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:15.747496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efef7b0c-d2ef-45b6-a4e7-dae3dedf7bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:15.748173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efef7b0c-d2ef-45b6-a4e7-dae3dedf7bf4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.748457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efef7b0c-d2ef-45b6-a4e7-dae3dedf7bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.748731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efef7b0c-d2ef-45b6-a4e7-dae3dedf7bf4 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.750061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efef7b0c-d2ef-45b6-a4e7-dae3dedf7bf4 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:15.750370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efef7b0c-d2ef-45b6-a4e7-dae3dedf7bf4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:15.751817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efef7b0c-d2ef-45b6-a4e7-dae3dedf7bf4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:15.777074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 398/793] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Jun 10 12:46:15 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:15.777859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efef7b0c-d2ef-45b6-a4e7-dae3dedf7bf4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:15.873058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86d6c61-9aa4-4e3c-ab24-5b1c5bf69e92 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:15.880750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:15.887975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efef7b0c-d2ef-45b6-a4e7-dae3dedf7bf4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:15.918349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db497366-efd7-42f3-bda7-69ff26cee5b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:15.919696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db497366-efd7-42f3-bda7-69ff26cee5b2 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.919988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db497366-efd7-42f3-bda7-69ff26cee5b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.920269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db497366-efd7-42f3-bda7-69ff26cee5b2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.926747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a2bc2c-c806-4347-9f72-6be1d3fc33fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:15.927644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a2bc2c-c806-4347-9f72-6be1d3fc33fe None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:15.928088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a2bc2c-c806-4347-9f72-6be1d3fc33fe None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:15.928531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a2bc2c-c806-4347-9f72-6be1d3fc33fe None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:15.929533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a2bc2c-c806-4347-9f72-6be1d3fc33fe None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:15.929984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a2bc2c-c806-4347-9f72-6be1d3fc33fe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:15.931495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a2bc2c-c806-4347-9f72-6be1d3fc33fe None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:15.932917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db497366-efd7-42f3-bda7-69ff26cee5b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-db497366-efd7-42f3-bda7-69ff26cee5b2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:15.944851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:15.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 396/794] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:15.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db497366-efd7-42f3-bda7-69ff26cee5b2 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:15.948255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 399/795] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:15.950020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:15.951488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a2bc2c-c806-4347-9f72-6be1d3fc33fe None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.067254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a2bc2c-c806-4347-9f72-6be1d3fc33fe None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.072697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db497366-efd7-42f3-bda7-69ff26cee5b2 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.109277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a22728b-fd66-4cde-9ca7-331ffd364b0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.110442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a22728b-fd66-4cde-9ca7-331ffd364b0f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.110941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a22728b-fd66-4cde-9ca7-331ffd364b0f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.111422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a22728b-fd66-4cde-9ca7-331ffd364b0f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.114037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afc2776-f7e4-4d6d-b777-e28513d3a6f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.115236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afc2776-f7e4-4d6d-b777-e28513d3a6f6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.115525 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afc2776-f7e4-4d6d-b777-e28513d3a6f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.116338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afc2776-f7e4-4d6d-b777-e28513d3a6f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.124885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a22728b-fd66-4cde-9ca7-331ffd364b0f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.126706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a22728b-fd66-4cde-9ca7-331ffd364b0f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:16.129865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afc2776-f7e4-4d6d-b777-e28513d3a6f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.131324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afc2776-f7e4-4d6d-b777-e28513d3a6f6 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-5a22728b-fd66-4cde-9ca7-331ffd364b0f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:16.146103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:16.148001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:16.148933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:16.148933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:16.148933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:16.148933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-7afc2776-f7e4-4d6d-b777-e28513d3a6f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:16.158281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:16.159167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:16.160082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:16.160082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:16.160082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:16.160082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:16.160082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 400/796] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:16.160082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afc2776-f7e4-4d6d-b777-e28513d3a6f6 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.161146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 397/797] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:16.166839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a22728b-fd66-4cde-9ca7-331ffd364b0f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.168506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:16.237982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:16.277827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afc2776-f7e4-4d6d-b777-e28513d3a6f6 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.296811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a22728b-fd66-4cde-9ca7-331ffd364b0f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.316969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540db6ca-2456-4a71-bef4-36cd581faa7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.318035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540db6ca-2456-4a71-bef4-36cd581faa7c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.318625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540db6ca-2456-4a71-bef4-36cd581faa7c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.319673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540db6ca-2456-4a71-bef4-36cd581faa7c None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.320635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540db6ca-2456-4a71-bef4-36cd581faa7c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:16.321182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540db6ca-2456-4a71-bef4-36cd581faa7c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.322845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540db6ca-2456-4a71-bef4-36cd581faa7c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:16.341714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 401/798] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:16.341714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540db6ca-2456-4a71-bef4-36cd581faa7c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.353118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f40f97-c84d-4729-baf1-85b8326f85c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.353791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f40f97-c84d-4729-baf1-85b8326f85c7 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.354077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f40f97-c84d-4729-baf1-85b8326f85c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.354354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f40f97-c84d-4729-baf1-85b8326f85c7 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.355386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f40f97-c84d-4729-baf1-85b8326f85c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:16.355386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f40f97-c84d-4729-baf1-85b8326f85c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.356920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f40f97-c84d-4729-baf1-85b8326f85c7 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:16.374866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 398/799] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:16.375796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f40f97-c84d-4729-baf1-85b8326f85c7 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.441975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:16.484404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540db6ca-2456-4a71-bef4-36cd581faa7c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.488865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f40f97-c84d-4729-baf1-85b8326f85c7 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.507064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:16.524716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22d6b6e-4c24-4eae-99f0-747a263f1a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.525530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22d6b6e-4c24-4eae-99f0-747a263f1a79 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.526063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22d6b6e-4c24-4eae-99f0-747a263f1a79 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.526347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22d6b6e-4c24-4eae-99f0-747a263f1a79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.527407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2595ad-3979-4d75-b7ee-cdcf68a8a4f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.528072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2595ad-3979-4d75-b7ee-cdcf68a8a4f1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.528352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2595ad-3979-4d75-b7ee-cdcf68a8a4f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.528626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2595ad-3979-4d75-b7ee-cdcf68a8a4f1 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.529364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2595ad-3979-4d75-b7ee-cdcf68a8a4f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:16.529735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2595ad-3979-4d75-b7ee-cdcf68a8a4f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.531080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2595ad-3979-4d75-b7ee-cdcf68a8a4f1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:16.536512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22d6b6e-4c24-4eae-99f0-747a263f1a79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.538540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22d6b6e-4c24-4eae-99f0-747a263f1a79 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:16.549205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 402/800] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:16.549205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2595ad-3979-4d75-b7ee-cdcf68a8a4f1 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-c22d6b6e-4c24-4eae-99f0-747a263f1a79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:16.561621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:16.562535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:16.563458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:16.563458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:16.563458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:16.563458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:16.563458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 399/801] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:16.564174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7707bb0-ad02-4235-a5e2-0935dcd7ee93 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.564469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7707bb0-ad02-4235-a5e2-0935dcd7ee93 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.564651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7707bb0-ad02-4235-a5e2-0935dcd7ee93 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.565922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 400/802] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:16.566613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7707bb0-ad02-4235-a5e2-0935dcd7ee93 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.655363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2595ad-3979-4d75-b7ee-cdcf68a8a4f1 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.673896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7707bb0-ad02-4235-a5e2-0935dcd7ee93 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.687259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:16.692184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29b8aea-4144-4ced-8a5e-22bf4bbc449f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.692996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29b8aea-4144-4ced-8a5e-22bf4bbc449f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.693283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29b8aea-4144-4ced-8a5e-22bf4bbc449f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.693558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29b8aea-4144-4ced-8a5e-22bf4bbc449f None admin] PATH_INFO: `/v3/users/swift` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.700611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29b8aea-4144-4ced-8a5e-22bf4bbc449f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.702091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29b8aea-4144-4ced-8a5e-22bf4bbc449f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:16.708998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314730d0-8ef3-4519-8ed4-79d8670bf4a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.710405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314730d0-8ef3-4519-8ed4-79d8670bf4a8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.710684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314730d0-8ef3-4519-8ed4-79d8670bf4a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.710960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314730d0-8ef3-4519-8ed4-79d8670bf4a8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-f29b8aea-4144-4ced-8a5e-22bf4bbc449f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:16.717152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:16.718068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:46:16.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:16.719562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314730d0-8ef3-4519-8ed4-79d8670bf4a8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.719562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314730d0-8ef3-4519-8ed4-79d8670bf4a8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:16.723078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 403/803] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:16 2022] GET /identity/v3/users/swift => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:16.723676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29b8aea-4144-4ced-8a5e-22bf4bbc449f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-314730d0-8ef3-4519-8ed4-79d8670bf4a8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:16.734528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:16.735451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 401/804] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fd5b2-50dd-4b69-9412-4c0c6d9beada None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.741039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fd5b2-50dd-4b69-9412-4c0c6d9beada None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.741669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011fd5b2-50dd-4b69-9412-4c0c6d9beada None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.755751 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-011fd5b2-50dd-4b69-9412-4c0c6d9beada None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:16.757165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-011fd5b2-50dd-4b69-9412-4c0c6d9beada None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:16.780466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:16.828233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 402/805] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:16.829016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011fd5b2-50dd-4b69-9412-4c0c6d9beada None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.842334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29b8aea-4144-4ced-8a5e-22bf4bbc449f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.880594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6133169e-621f-479f-9619-2c913fffcc48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.881337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6133169e-621f-479f-9619-2c913fffcc48 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.882160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6133169e-621f-479f-9619-2c913fffcc48 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.882932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6133169e-621f-479f-9619-2c913fffcc48 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.883176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6133169e-621f-479f-9619-2c913fffcc48 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:16.883485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6133169e-621f-479f-9619-2c913fffcc48 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.884915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6133169e-621f-479f-9619-2c913fffcc48 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:16.903537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 404/806] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:16.903537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6133169e-621f-479f-9619-2c913fffcc48 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:16.950963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011fd5b2-50dd-4b69-9412-4c0c6d9beada None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:16.956884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:16.984839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa420a08-eb51-4335-b031-5e91b724ee86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:16.985511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa420a08-eb51-4335-b031-5e91b724ee86 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:16.985797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa420a08-eb51-4335-b031-5e91b724ee86 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:16.986078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa420a08-eb51-4335-b031-5e91b724ee86 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:16.986968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa420a08-eb51-4335-b031-5e91b724ee86 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:16.987281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa420a08-eb51-4335-b031-5e91b724ee86 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:16.988723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa420a08-eb51-4335-b031-5e91b724ee86 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.002069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6133169e-621f-479f-9619-2c913fffcc48 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.005121 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 403/807] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Jun 10 12:46:16 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:17.005914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa420a08-eb51-4335-b031-5e91b724ee86 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.024389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:17.036568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965e1ec0-ab1d-4e13-a590-5902fb575cee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.037270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965e1ec0-ab1d-4e13-a590-5902fb575cee None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.037552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965e1ec0-ab1d-4e13-a590-5902fb575cee None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.037864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965e1ec0-ab1d-4e13-a590-5902fb575cee None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.038649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965e1ec0-ab1d-4e13-a590-5902fb575cee None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:17.038951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965e1ec0-ab1d-4e13-a590-5902fb575cee None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.040382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-965e1ec0-ab1d-4e13-a590-5902fb575cee None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.052033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 405/808] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:17.052840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965e1ec0-ab1d-4e13-a590-5902fb575cee None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.109670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa420a08-eb51-4335-b031-5e91b724ee86 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.142245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94ef444-3752-4d06-948a-d677fbf26038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.143037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ef444-3752-4d06-948a-d677fbf26038 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.143316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ef444-3752-4d06-948a-d677fbf26038 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.143594 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ef444-3752-4d06-948a-d677fbf26038 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.144229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ef444-3752-4d06-948a-d677fbf26038 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.145619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ef444-3752-4d06-948a-d677fbf26038 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.149744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965e1ec0-ab1d-4e13-a590-5902fb575cee None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-f94ef444-3752-4d06-948a-d677fbf26038 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:17.158194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:17.159324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:17.160263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 404/809] 10.222.0.25 () {56 vars in 1129 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:17.161137 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce200d-16ba-48e7-82bf-e87852ebef7f None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.161137 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce200d-16ba-48e7-82bf-e87852ebef7f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.161137 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abce200d-16ba-48e7-82bf-e87852ebef7f None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.162478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 405/810] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:17 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:17.163974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abce200d-16ba-48e7-82bf-e87852ebef7f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.166277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:17.187482 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9e72cc-b569-4144-bbff-b15c021cc859 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.188262 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9e72cc-b569-4144-bbff-b15c021cc859 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.188544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9e72cc-b569-4144-bbff-b15c021cc859 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.188849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9e72cc-b569-4144-bbff-b15c021cc859 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.196186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9e72cc-b569-4144-bbff-b15c021cc859 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.197780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9e72cc-b569-4144-bbff-b15c021cc859 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-9f9e72cc-b569-4144-bbff-b15c021cc859 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:17.213204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:17.214157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:17.215078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 406/811] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/users/nova => generated 82 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:17.215653 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9e72cc-b569-4144-bbff-b15c021cc859 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.243368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:17.262540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abce200d-16ba-48e7-82bf-e87852ebef7f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.295576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28c9242-208c-4cec-9222-0c026100b18b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.296351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28c9242-208c-4cec-9222-0c026100b18b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.296633 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28c9242-208c-4cec-9222-0c026100b18b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.296942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28c9242-208c-4cec-9222-0c026100b18b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.306595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28c9242-208c-4cec-9222-0c026100b18b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.308030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28c9242-208c-4cec-9222-0c026100b18b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.315970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9e72cc-b569-4144-bbff-b15c021cc859 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-b28c9242-208c-4cec-9222-0c026100b18b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:17.327146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:17.328146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:17.329080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:17.329080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:17.329080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:17.329080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:17.329080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 406/812] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:17.329354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28c9242-208c-4cec-9222-0c026100b18b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.353180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0513109-c954-4e4a-9af1-c239ede4cd2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.354062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0513109-c954-4e4a-9af1-c239ede4cd2c None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.354353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0513109-c954-4e4a-9af1-c239ede4cd2c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.354609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0513109-c954-4e4a-9af1-c239ede4cd2c None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/eb2c6f06b300473ebafff4503bb24396/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.368195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0513109-c954-4e4a-9af1-c239ede4cd2c None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=eb2c6f06b300473ebafff4503bb24396, role_id=c7c592df9b6b4743a1ba6ca3b2a7fdbc)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.369654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0513109-c954-4e4a-9af1-c239ede4cd2c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.429903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28c9242-208c-4cec-9222-0c026100b18b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.437735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 407/813] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:46:17 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/eb2c6f06b300473ebafff4503bb24396/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:46:17.439091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890039aa-6de6-4a22-b465-8edfd928f746 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.439385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890039aa-6de6-4a22-b465-8edfd928f746 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.439661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890039aa-6de6-4a22-b465-8edfd928f746 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.469605 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:17.469819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-890039aa-6de6-4a22-b465-8edfd928f746 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:17.469819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-890039aa-6de6-4a22-b465-8edfd928f746 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:17.470528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc7c637-a2ab-4ada-93f2-ff91a483169a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.471208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc7c637-a2ab-4ada-93f2-ff91a483169a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.471492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc7c637-a2ab-4ada-93f2-ff91a483169a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.471765 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc7c637-a2ab-4ada-93f2-ff91a483169a None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.472474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc7c637-a2ab-4ada-93f2-ff91a483169a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:17.472793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc7c637-a2ab-4ada-93f2-ff91a483169a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.474288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc7c637-a2ab-4ada-93f2-ff91a483169a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.484388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 407/814] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:17.484388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc7c637-a2ab-4ada-93f2-ff91a483169a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.487354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:17.548837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 408/815] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:17 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:17.550087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890039aa-6de6-4a22-b465-8edfd928f746 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.625096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc7c637-a2ab-4ada-93f2-ff91a483169a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.662271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8113165-e960-463f-af57-0d73690f2ac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.663543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8113165-e960-463f-af57-0d73690f2ac3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.663827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8113165-e960-463f-af57-0d73690f2ac3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.664100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8113165-e960-463f-af57-0d73690f2ac3 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.665049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8113165-e960-463f-af57-0d73690f2ac3 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:17.665354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8113165-e960-463f-af57-0d73690f2ac3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.666886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8113165-e960-463f-af57-0d73690f2ac3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.674729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-890039aa-6de6-4a22-b465-8edfd928f746 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.689568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 408/816] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:17.689568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8113165-e960-463f-af57-0d73690f2ac3 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.714432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc43282-b133-454e-b61b-c7184b0c3044 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.715315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc43282-b133-454e-b61b-c7184b0c3044 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.715758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc43282-b133-454e-b61b-c7184b0c3044 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.716033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc43282-b133-454e-b61b-c7184b0c3044 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.716748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc43282-b133-454e-b61b-c7184b0c3044 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:17.717081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc43282-b133-454e-b61b-c7184b0c3044 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.718445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc43282-b133-454e-b61b-c7184b0c3044 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.735832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 409/817] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:17.735832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc43282-b133-454e-b61b-c7184b0c3044 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.749247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:17.749531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:17.795914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8113165-e960-463f-af57-0d73690f2ac3 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.831950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ddee5e-4455-45b3-ac3b-6726d1eddf02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.832843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ddee5e-4455-45b3-ac3b-6726d1eddf02 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.833135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ddee5e-4455-45b3-ac3b-6726d1eddf02 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.833415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ddee5e-4455-45b3-ac3b-6726d1eddf02 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.834112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ddee5e-4455-45b3-ac3b-6726d1eddf02 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:17.834929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ddee5e-4455-45b3-ac3b-6726d1eddf02 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.836301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ddee5e-4455-45b3-ac3b-6726d1eddf02 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.838475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc43282-b133-454e-b61b-c7184b0c3044 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.845611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 409/818] 10.222.0.25 () {56 vars in 1124 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:17.846462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ddee5e-4455-45b3-ac3b-6726d1eddf02 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.874862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b9aa06-229a-4263-9cda-fed86c0e0ddd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.875928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b9aa06-229a-4263-9cda-fed86c0e0ddd None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.876398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b9aa06-229a-4263-9cda-fed86c0e0ddd None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.876668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b9aa06-229a-4263-9cda-fed86c0e0ddd None admin] PATH_INFO: `/v3/services/network` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:17.877347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b9aa06-229a-4263-9cda-fed86c0e0ddd None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:17.878714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b9aa06-229a-4263-9cda-fed86c0e0ddd None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-35b9aa06-229a-4263-9cda-fed86c0e0ddd None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:17.891397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:17.892448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:17.893320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 410/819] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/services/network => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:17.894187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b9aa06-229a-4263-9cda-fed86c0e0ddd None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:17.958224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ddee5e-4455-45b3-ac3b-6726d1eddf02 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:17.977870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:17.980712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:17.995808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c92c283-0807-4911-9988-8f0ace5b8cc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:17.996406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c92c283-0807-4911-9988-8f0ace5b8cc1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:17.996694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c92c283-0807-4911-9988-8f0ace5b8cc1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:17.997013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c92c283-0807-4911-9988-8f0ace5b8cc1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:18.006882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b9aa06-229a-4263-9cda-fed86c0e0ddd None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:18.041847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77aede7e-20ba-401e-a892-d4c169c8e01d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:18.042646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77aede7e-20ba-401e-a892-d4c169c8e01d None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:18.042932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77aede7e-20ba-401e-a892-d4c169c8e01d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:18.043209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77aede7e-20ba-401e-a892-d4c169c8e01d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:18.050980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77aede7e-20ba-401e-a892-d4c169c8e01d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:18.052462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77aede7e-20ba-401e-a892-d4c169c8e01d None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:18.066189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c92c283-0807-4911-9988-8f0ace5b8cc1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:18.067636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c92c283-0807-4911-9988-8f0ace5b8cc1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-77aede7e-20ba-401e-a892-d4c169c8e01d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:18.067963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:18.069395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:18.070293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:18.070293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:18.070293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:18.070293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:18.070293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:18.070293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:18.070293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:18.070293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:18.070293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 411/820] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:18.071549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77aede7e-20ba-401e-a892-d4c169c8e01d None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:18.175246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77aede7e-20ba-401e-a892-d4c169c8e01d None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:18.176651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 410/821] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:17 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:46:18.181715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c92c283-0807-4911-9988-8f0ace5b8cc1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:18.204152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:18.211296 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b87f7ac-8d7f-431d-a7dc-1c712eaa8e47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:18.211953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b87f7ac-8d7f-431d-a7dc-1c712eaa8e47 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:18.212236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b87f7ac-8d7f-431d-a7dc-1c712eaa8e47 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:18.212514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b87f7ac-8d7f-431d-a7dc-1c712eaa8e47 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:18.213291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b87f7ac-8d7f-431d-a7dc-1c712eaa8e47 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:18.217959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b87f7ac-8d7f-431d-a7dc-1c712eaa8e47 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:18.220259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:18.288935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c92c283-0807-4911-9988-8f0ace5b8cc1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:18.309481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 412/822] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Jun 10 12:46:18 2022] POST /identity/v3/endpoints => generated 346 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:46:18.310483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b87f7ac-8d7f-431d-a7dc-1c712eaa8e47 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:18.358382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdca623-903b-4aad-b84d-1e13d652b3c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:18.360952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdca623-903b-4aad-b84d-1e13d652b3c8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:18.361551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdca623-903b-4aad-b84d-1e13d652b3c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:18.362036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfdca623-903b-4aad-b84d-1e13d652b3c8 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:18.363005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdca623-903b-4aad-b84d-1e13d652b3c8 None admin] RBAC: Adding query filter params (name=network) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:18.363534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdca623-903b-4aad-b84d-1e13d652b3c8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:18.365234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfdca623-903b-4aad-b84d-1e13d652b3c8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:18.373559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 411/823] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:18 2022] GET /identity/v3/services?name=network => generated 124 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:18.375003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdca623-903b-4aad-b84d-1e13d652b3c8 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:18.436719 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b87f7ac-8d7f-431d-a7dc-1c712eaa8e47 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:18.474417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb2af33-6a8e-46e5-9838-7a1674511571 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:18.475116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb2af33-6a8e-46e5-9838-7a1674511571 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:18.475401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb2af33-6a8e-46e5-9838-7a1674511571 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:18.475679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb2af33-6a8e-46e5-9838-7a1674511571 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:18.476524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb2af33-6a8e-46e5-9838-7a1674511571 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:18.476864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb2af33-6a8e-46e5-9838-7a1674511571 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:18.478358 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb2af33-6a8e-46e5-9838-7a1674511571 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:18.485185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:18.493954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 413/824] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:18.494784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb2af33-6a8e-46e5-9838-7a1674511571 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:18.519614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfdca623-903b-4aad-b84d-1e13d652b3c8 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:18.561323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:18.573946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615afe32-409c-4c5a-8960-9547b2529f34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:18.574740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615afe32-409c-4c5a-8960-9547b2529f34 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:18.575016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615afe32-409c-4c5a-8960-9547b2529f34 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:18.575291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-615afe32-409c-4c5a-8960-9547b2529f34 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:18.591277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615afe32-409c-4c5a-8960-9547b2529f34 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:18.593507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-615afe32-409c-4c5a-8960-9547b2529f34 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:18.615053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb2af33-6a8e-46e5-9838-7a1674511571 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-615afe32-409c-4c5a-8960-9547b2529f34 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:18.616892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:18.617768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:18.618661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:18.618661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:18.618661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:18.618661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:18.619504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 412/825] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:18.620033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615afe32-409c-4c5a-8960-9547b2529f34 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:18.660040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed626ab-e986-4e14-858a-94a8098eb3f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:18.660988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed626ab-e986-4e14-858a-94a8098eb3f6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:18.662197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed626ab-e986-4e14-858a-94a8098eb3f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:18.668794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed626ab-e986-4e14-858a-94a8098eb3f6 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:18.668794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed626ab-e986-4e14-858a-94a8098eb3f6 None admin] RBAC: Adding query filter params (type=network) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:18.668794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed626ab-e986-4e14-858a-94a8098eb3f6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:18.668794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed626ab-e986-4e14-858a-94a8098eb3f6 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:18.672884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 414/826] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:18 2022] GET /identity/v3/services?type=network => generated 350 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:18.673732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed626ab-e986-4e14-858a-94a8098eb3f6 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:18.753996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-615afe32-409c-4c5a-8960-9547b2529f34 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:18.781879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed626ab-e986-4e14-858a-94a8098eb3f6 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:18.796996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:18.799586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:18.804307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7c3d4d-846d-4667-ba0c-97ac76376016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:18.805487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7c3d4d-846d-4667-ba0c-97ac76376016 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:18.805950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7c3d4d-846d-4667-ba0c-97ac76376016 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:18.811424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7c3d4d-846d-4667-ba0c-97ac76376016 None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/bb18cd41c9a14d358aac14ed62ef9e02/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:18.811424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7c3d4d-846d-4667-ba0c-97ac76376016 None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=bb18cd41c9a14d358aac14ed62ef9e02, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:18.811424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7c3d4d-846d-4667-ba0c-97ac76376016 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:18.818590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17bd3f1-531a-4283-88f8-f5896199075a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:18.819809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17bd3f1-531a-4283-88f8-f5896199075a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:18.820236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17bd3f1-531a-4283-88f8-f5896199075a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:18.820523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17bd3f1-531a-4283-88f8-f5896199075a None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:18.824728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17bd3f1-531a-4283-88f8-f5896199075a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:18.825925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17bd3f1-531a-4283-88f8-f5896199075a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:18.827699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f17bd3f1-531a-4283-88f8-f5896199075a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:18.845143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 415/827] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:18.850253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17bd3f1-531a-4283-88f8-f5896199075a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:18.873290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 413/828] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:46:18 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/bb18cd41c9a14d358aac14ed62ef9e02/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:46:18.874780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7c3d4d-846d-4667-ba0c-97ac76376016 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:19.007303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7c3d4d-846d-4667-ba0c-97ac76376016 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:19.049824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a756a8c-c112-41ab-9403-e603a9467b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:19.051183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a756a8c-c112-41ab-9403-e603a9467b57 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:19.051675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a756a8c-c112-41ab-9403-e603a9467b57 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:19.052502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a756a8c-c112-41ab-9403-e603a9467b57 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:19.062598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a756a8c-c112-41ab-9403-e603a9467b57 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:19.065151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a756a8c-c112-41ab-9403-e603a9467b57 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:19.073956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f17bd3f1-531a-4283-88f8-f5896199075a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-5a756a8c-c112-41ab-9403-e603a9467b57 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:19.083078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:19.083996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:19.085467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:19.085467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:19.085467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:19.085467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:19.085467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:19.085467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:19.085467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:19.085467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:19.088944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 414/829] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:19.094748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a756a8c-c112-41ab-9403-e603a9467b57 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:19.119408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bbc763-ef67-4f9f-bad7-6dc65dbd4f0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:19.120039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bbc763-ef67-4f9f-bad7-6dc65dbd4f0d None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:19.120322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bbc763-ef67-4f9f-bad7-6dc65dbd4f0d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:19.120604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3bbc763-ef67-4f9f-bad7-6dc65dbd4f0d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:19.121378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bbc763-ef67-4f9f-bad7-6dc65dbd4f0d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:19.122976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3bbc763-ef67-4f9f-bad7-6dc65dbd4f0d None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:19.161107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:19.216807 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 416/830] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Jun 10 12:46:18 2022] POST /identity/v3/endpoints => generated 327 bytes in 371 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:46:19.220368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a756a8c-c112-41ab-9403-e603a9467b57 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:19.265054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2642e36d-a181-48b8-9c07-e6bd17a18b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:19.265796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2642e36d-a181-48b8-9c07-e6bd17a18b67 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:19.266082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2642e36d-a181-48b8-9c07-e6bd17a18b67 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:19.266362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2642e36d-a181-48b8-9c07-e6bd17a18b67 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:19.267165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2642e36d-a181-48b8-9c07-e6bd17a18b67 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:19.267476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2642e36d-a181-48b8-9c07-e6bd17a18b67 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:19.269077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2642e36d-a181-48b8-9c07-e6bd17a18b67 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:19.284864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 415/831] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:19.291326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bbc763-ef67-4f9f-bad7-6dc65dbd4f0d None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:19.366148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:19.460941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3bbc763-ef67-4f9f-bad7-6dc65dbd4f0d None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:19.500894 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7cc451c-b772-4df6-b2cd-36a4a8cfbfba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:19.501549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cc451c-b772-4df6-b2cd-36a4a8cfbfba None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:19.501831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cc451c-b772-4df6-b2cd-36a4a8cfbfba None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:19.502103 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7cc451c-b772-4df6-b2cd-36a4a8cfbfba None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:19.502935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cc451c-b772-4df6-b2cd-36a4a8cfbfba None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=e9a0496199134369a0ef740c099eed59) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:19.503245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cc451c-b772-4df6-b2cd-36a4a8cfbfba None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:19.504669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7cc451c-b772-4df6-b2cd-36a4a8cfbfba None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:19.513451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 417/832] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:19 2022] GET /identity/v3/role_assignments?user.id=e9a0496199134369a0ef740c099eed59&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=766799d1b6c049168590571f1a959b3b => generated 608 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:19.788498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c123d-ed57-4b76-8166-a623b3bc26d0 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:19.789044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c123d-ed57-4b76-8166-a623b3bc26d0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:19.789145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c123d-ed57-4b76-8166-a623b3bc26d0 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:19.790735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 416/833] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:19.795865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc9be5-e875-463d-98e8-a0f8fa0d8c23 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:19.796376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc9be5-e875-463d-98e8-a0f8fa0d8c23 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:19.796915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bc9be5-e875-463d-98e8-a0f8fa0d8c23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:19.814433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-00bc9be5-e875-463d-98e8-a0f8fa0d8c23 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:19.815920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-00bc9be5-e875-463d-98e8-a0f8fa0d8c23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:19.886386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 418/834] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:19 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:19.910175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c123d-ed57-4b76-8166-a623b3bc26d0 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:20.142444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c123d-ed57-4b76-8166-a623b3bc26d0 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:20.154303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:20.180379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a129b8-7458-469c-919e-aa30709538e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:20.181202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a129b8-7458-469c-919e-aa30709538e8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:20.181485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a129b8-7458-469c-919e-aa30709538e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:20.181761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a129b8-7458-469c-919e-aa30709538e8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:20.192206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a129b8-7458-469c-919e-aa30709538e8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-e6a129b8-7458-469c-919e-aa30709538e8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:20.199923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:20.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:20.201754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 417/835] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:46:19 2022] GET /identity/v3/roles/service => generated 85 bytes in 292 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:20.208793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc9be5-e875-463d-98e8-a0f8fa0d8c23 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:20.238756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:20.312320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bc9be5-e875-463d-98e8-a0f8fa0d8c23 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:20.347200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf976233-eab4-4724-9e20-cca97b116a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:20.347930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf976233-eab4-4724-9e20-cca97b116a69 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:20.348215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf976233-eab4-4724-9e20-cca97b116a69 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:20.348535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf976233-eab4-4724-9e20-cca97b116a69 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:20.349341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf976233-eab4-4724-9e20-cca97b116a69 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:20.349656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf976233-eab4-4724-9e20-cca97b116a69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:20.351553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf976233-eab4-4724-9e20-cca97b116a69 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:20.363217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 419/836] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:46:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:20.369016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a129b8-7458-469c-919e-aa30709538e8 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:20.488334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a129b8-7458-469c-919e-aa30709538e8 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:20.523754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46f15be-5dd7-4c81-9441-de12f811379c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:20.524545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46f15be-5dd7-4c81-9441-de12f811379c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:20.524867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46f15be-5dd7-4c81-9441-de12f811379c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:20.525149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46f15be-5dd7-4c81-9441-de12f811379c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:20.535672 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46f15be-5dd7-4c81-9441-de12f811379c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:20.537154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46f15be-5dd7-4c81-9441-de12f811379c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-c46f15be-5dd7-4c81-9441-de12f811379c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:20.555054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:20.556101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:20.557221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:20.557221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:20.557221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:20.557221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:20.557221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 418/837] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:20.562140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf976233-eab4-4724-9e20-cca97b116a69 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:20.663391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:20.676036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf976233-eab4-4724-9e20-cca97b116a69 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:20.710047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a277d5e9-e97f-4432-a862-57f243a04c74 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:20.710361 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a277d5e9-e97f-4432-a862-57f243a04c74 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:20.710717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a277d5e9-e97f-4432-a862-57f243a04c74 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:20.712394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 419/838] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:20.717246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b89f053-f6c1-4553-81fe-60bde5ccc51f None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:20.717437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf61c7b-b640-46cd-90aa-b38fe1279d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:20.717616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b89f053-f6c1-4553-81fe-60bde5ccc51f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:20.717785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b89f053-f6c1-4553-81fe-60bde5ccc51f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:20.717987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf61c7b-b640-46cd-90aa-b38fe1279d42 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:20.718271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf61c7b-b640-46cd-90aa-b38fe1279d42 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:20.718546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf61c7b-b640-46cd-90aa-b38fe1279d42 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:20.719268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf61c7b-b640-46cd-90aa-b38fe1279d42 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:20.719571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf61c7b-b640-46cd-90aa-b38fe1279d42 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:20.721107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf61c7b-b640-46cd-90aa-b38fe1279d42 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:20.734077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 420/839] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:20.739131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1b89f053-f6c1-4553-81fe-60bde5ccc51f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:20.739327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf61c7b-b640-46cd-90aa-b38fe1279d42 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:20.740249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1b89f053-f6c1-4553-81fe-60bde5ccc51f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:20.753962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:20.816128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 420/840] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:20 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:20.839055 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b89f053-f6c1-4553-81fe-60bde5ccc51f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:20.864817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf61c7b-b640-46cd-90aa-b38fe1279d42 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:20.906546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28668bc0-aa07-43b9-97ca-ddcd19ec4a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:20.907357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28668bc0-aa07-43b9-97ca-ddcd19ec4a84 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:20.907646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28668bc0-aa07-43b9-97ca-ddcd19ec4a84 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:20.907931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28668bc0-aa07-43b9-97ca-ddcd19ec4a84 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:20.919806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28668bc0-aa07-43b9-97ca-ddcd19ec4a84 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:20.921618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28668bc0-aa07-43b9-97ca-ddcd19ec4a84 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-28668bc0-aa07-43b9-97ca-ddcd19ec4a84 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:20.937940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:20.938923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:20.939887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:20.944465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 421/841] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:46:20 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:20.944816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b89f053-f6c1-4553-81fe-60bde5ccc51f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:20.945445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28668bc0-aa07-43b9-97ca-ddcd19ec4a84 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:20.957305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:20.980248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0612df55-ed28-4964-85e0-bfc2021d76ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:20.981127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0612df55-ed28-4964-85e0-bfc2021d76ef None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:20.981413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0612df55-ed28-4964-85e0-bfc2021d76ef None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:20.981691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0612df55-ed28-4964-85e0-bfc2021d76ef None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:20.982377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0612df55-ed28-4964-85e0-bfc2021d76ef None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:20.983702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0612df55-ed28-4964-85e0-bfc2021d76ef None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-0612df55-ed28-4964-85e0-bfc2021d76ef None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:20.995707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:20.996961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:20.997907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 421/842] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:46:20 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:21.002754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0612df55-ed28-4964-85e0-bfc2021d76ef None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:21.018048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:21.065283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28668bc0-aa07-43b9-97ca-ddcd19ec4a84 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:21.103834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b9fa03-e688-4f59-91ff-793a530f52a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:21.104811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9fa03-e688-4f59-91ff-793a530f52a7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.105206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0612df55-ed28-4964-85e0-bfc2021d76ef None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:21.105400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9fa03-e688-4f59-91ff-793a530f52a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.105816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b9fa03-e688-4f59-91ff-793a530f52a7 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.106911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b9fa03-e688-4f59-91ff-793a530f52a7 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:21.107468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b9fa03-e688-4f59-91ff-793a530f52a7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:21.109182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b9fa03-e688-4f59-91ff-793a530f52a7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:21.133112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 422/843] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Jun 10 12:46:20 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:21.135271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b9fa03-e688-4f59-91ff-793a530f52a7 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:21.140782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52490df3-c835-4242-b358-195f9490eb57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:21.141514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52490df3-c835-4242-b358-195f9490eb57 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.141805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52490df3-c835-4242-b358-195f9490eb57 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.142127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52490df3-c835-4242-b358-195f9490eb57 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.142837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52490df3-c835-4242-b358-195f9490eb57 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:21.143144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52490df3-c835-4242-b358-195f9490eb57 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:21.144516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52490df3-c835-4242-b358-195f9490eb57 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:21.154813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 422/844] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:46:21 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:21.155941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52490df3-c835-4242-b358-195f9490eb57 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:21.207461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:21.235689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:21.247374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b9fa03-e688-4f59-91ff-793a530f52a7 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:21.262865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52490df3-c835-4242-b358-195f9490eb57 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:21.289488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5543a433-dfe1-4e8f-8486-5ca2a064f1b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:21.290420 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5543a433-dfe1-4e8f-8486-5ca2a064f1b5 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.290853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5543a433-dfe1-4e8f-8486-5ca2a064f1b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.291302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5543a433-dfe1-4e8f-8486-5ca2a064f1b5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.301172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4083bbc-3936-489b-b42d-2231c4c1a8ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:21.301484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4083bbc-3936-489b-b42d-2231c4c1a8ee None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.301768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4083bbc-3936-489b-b42d-2231c4c1a8ee None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.302044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4083bbc-3936-489b-b42d-2231c4c1a8ee None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.302749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4083bbc-3936-489b-b42d-2231c4c1a8ee None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:21.303598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4083bbc-3936-489b-b42d-2231c4c1a8ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:21.304989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4083bbc-3936-489b-b42d-2231c4c1a8ee None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:21.315001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 423/845] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:46:21 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:21.323310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf71249-0fb4-4912-9e62-d5657ada2a44 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.323643 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf71249-0fb4-4912-9e62-d5657ada2a44 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.323884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf71249-0fb4-4912-9e62-d5657ada2a44 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.325437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 424/846] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:21.336225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efae370-0f84-483a-9b9e-7123b4bdd29e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.336519 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efae370-0f84-483a-9b9e-7123b4bdd29e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.336819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6efae370-0f84-483a-9b9e-7123b4bdd29e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.363385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-6efae370-0f84-483a-9b9e-7123b4bdd29e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:21.366549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6efae370-0f84-483a-9b9e-7123b4bdd29e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:21.381111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5543a433-dfe1-4e8f-8486-5ca2a064f1b5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:21.383186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5543a433-dfe1-4e8f-8486-5ca2a064f1b5 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:21.464736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:21.472588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 425/847] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:21 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:21.495780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efae370-0f84-483a-9b9e-7123b4bdd29e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:21.515127 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 423/848] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:21 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:46:21.521251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5543a433-dfe1-4e8f-8486-5ca2a064f1b5 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:21.574803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:21.612556 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6efae370-0f84-483a-9b9e-7123b4bdd29e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:21.657259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5543a433-dfe1-4e8f-8486-5ca2a064f1b5 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:21.663058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd2ea18-df11-4546-9c28-2f7ad39e9488 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:21.664077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd2ea18-df11-4546-9c28-2f7ad39e9488 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.664546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd2ea18-df11-4546-9c28-2f7ad39e9488 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.665057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd2ea18-df11-4546-9c28-2f7ad39e9488 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.678902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd2ea18-df11-4546-9c28-2f7ad39e9488 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-ccd2ea18-df11-4546-9c28-2f7ad39e9488 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:21.689411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:21.690440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:21.691283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:21.695138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 426/849] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:21.700840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd2ea18-df11-4546-9c28-2f7ad39e9488 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:21.704277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dafc7db-8b24-49e7-8b48-4e0a451ead01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:21.706478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dafc7db-8b24-49e7-8b48-4e0a451ead01 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.707008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dafc7db-8b24-49e7-8b48-4e0a451ead01 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.707472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dafc7db-8b24-49e7-8b48-4e0a451ead01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.723797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dafc7db-8b24-49e7-8b48-4e0a451ead01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:21.725706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dafc7db-8b24-49e7-8b48-4e0a451ead01 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-3dafc7db-8b24-49e7-8b48-4e0a451ead01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:21.755826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:21.757397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:21.758308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:21.758308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:21.758308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:21.758308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:21.758548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 424/850] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 238 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:21.764251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b497057-2856-4148-aef3-61e48cc9050c None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.764536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b497057-2856-4148-aef3-61e48cc9050c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.764839 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b497057-2856-4148-aef3-61e48cc9050c None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.766318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 425/851] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:21.770430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b497057-2856-4148-aef3-61e48cc9050c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:21.790751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:21.821707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd2ea18-df11-4546-9c28-2f7ad39e9488 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:21.852655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:21.859112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a645720-cdb2-421b-8678-27497d06017b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:21.859772 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a645720-cdb2-421b-8678-27497d06017b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.860117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a645720-cdb2-421b-8678-27497d06017b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.860409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a645720-cdb2-421b-8678-27497d06017b None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.861193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a645720-cdb2-421b-8678-27497d06017b None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:21.861505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a645720-cdb2-421b-8678-27497d06017b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:21.862892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a645720-cdb2-421b-8678-27497d06017b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:21.872160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 427/852] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:21.873439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926f618-70d8-448d-93a1-70544219dbfc None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.873727 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926f618-70d8-448d-93a1-70544219dbfc None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.874005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1926f618-70d8-448d-93a1-70544219dbfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.882378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b497057-2856-4148-aef3-61e48cc9050c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:21.890176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1926f618-70d8-448d-93a1-70544219dbfc None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:21.891368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1926f618-70d8-448d-93a1-70544219dbfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:21.919691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a2f9f-b507-4945-bbcb-341d6b97c077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:21.920569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a2f9f-b507-4945-bbcb-341d6b97c077 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:21.921149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a2f9f-b507-4945-bbcb-341d6b97c077 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:21.921607 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559a2f9f-b507-4945-bbcb-341d6b97c077 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:21.922787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559a2f9f-b507-4945-bbcb-341d6b97c077 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:21.923240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559a2f9f-b507-4945-bbcb-341d6b97c077 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:21.925066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-559a2f9f-b507-4945-bbcb-341d6b97c077 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:21.937803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 426/853] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:21.938855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a2f9f-b507-4945-bbcb-341d6b97c077 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:21.964735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 428/854] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:21 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:21.965795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1926f618-70d8-448d-93a1-70544219dbfc None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.025123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:22.047368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559a2f9f-b507-4945-bbcb-341d6b97c077 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.077328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1926f618-70d8-448d-93a1-70544219dbfc None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.083806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cbe4f1-fa29-4334-87c3-ac6ab686005f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.084746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cbe4f1-fa29-4334-87c3-ac6ab686005f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.085055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cbe4f1-fa29-4334-87c3-ac6ab686005f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.085331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cbe4f1-fa29-4334-87c3-ac6ab686005f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.096858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cbe4f1-fa29-4334-87c3-ac6ab686005f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.098432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cbe4f1-fa29-4334-87c3-ac6ab686005f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.105436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:22.117477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de2b6e5-9786-4e67-82ea-47dd2503074b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.118274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de2b6e5-9786-4e67-82ea-47dd2503074b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.118612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de2b6e5-9786-4e67-82ea-47dd2503074b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c7cbe4f1-fa29-4334-87c3-ac6ab686005f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:22.118811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:22.119643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:22.120559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:22.120559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:22.120559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:22.120559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:22.120803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de2b6e5-9786-4e67-82ea-47dd2503074b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.120878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 427/855] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:22.120878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cbe4f1-fa29-4334-87c3-ac6ab686005f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.131230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de2b6e5-9786-4e67-82ea-47dd2503074b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.132702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de2b6e5-9786-4e67-82ea-47dd2503074b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-8de2b6e5-9786-4e67-82ea-47dd2503074b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:22.147884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:22.148822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:22.149736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:22.149736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:22.149736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:22.149736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:22.149736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:22.149736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:22.149736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:22.149736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:22.149736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 429/856] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:22.150210 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de2b6e5-9786-4e67-82ea-47dd2503074b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.234503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cbe4f1-fa29-4334-87c3-ac6ab686005f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.251968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de2b6e5-9786-4e67-82ea-47dd2503074b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.267597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:22.273656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce858ee-85d8-4386-bac8-7e0b8244b4cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.274462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce858ee-85d8-4386-bac8-7e0b8244b4cb None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.278808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce858ee-85d8-4386-bac8-7e0b8244b4cb None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.278808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce858ee-85d8-4386-bac8-7e0b8244b4cb None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.278808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce858ee-85d8-4386-bac8-7e0b8244b4cb None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.278808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bce858ee-85d8-4386-bac8-7e0b8244b4cb None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.288064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd25eae1-4ebe-498d-8d95-826afa793bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.288706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd25eae1-4ebe-498d-8d95-826afa793bdc None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.289011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd25eae1-4ebe-498d-8d95-826afa793bdc None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.289280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd25eae1-4ebe-498d-8d95-826afa793bdc None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.289984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd25eae1-4ebe-498d-8d95-826afa793bdc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:22.290284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd25eae1-4ebe-498d-8d95-826afa793bdc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-bce858ee-85d8-4386-bac8-7e0b8244b4cb None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:22.290623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:22.291492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jun 10 12:46:22.292369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:22.293809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd25eae1-4ebe-498d-8d95-826afa793bdc None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.293915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 428/857] 10.222.0.25 () {56 vars in 1132 bytes} [Fri Jun 10 12:46:22 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:22.293915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce858ee-85d8-4386-bac8-7e0b8244b4cb None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.311923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 430/858] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:22.311923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd25eae1-4ebe-498d-8d95-826afa793bdc None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.358088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:22.408937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce858ee-85d8-4386-bac8-7e0b8244b4cb None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.423410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd25eae1-4ebe-498d-8d95-826afa793bdc None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.445070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea701ab-b35a-4845-8740-35df155b83a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.445732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea701ab-b35a-4845-8740-35df155b83a1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.446016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea701ab-b35a-4845-8740-35df155b83a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.446319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea701ab-b35a-4845-8740-35df155b83a1 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.447077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea701ab-b35a-4845-8740-35df155b83a1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:22.447389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea701ab-b35a-4845-8740-35df155b83a1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.449096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea701ab-b35a-4845-8740-35df155b83a1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.459316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2288e7a-9ae9-4e3e-953e-0102cc9bc9bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.459999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2288e7a-9ae9-4e3e-953e-0102cc9bc9bf None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.460285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2288e7a-9ae9-4e3e-953e-0102cc9bc9bf None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.460561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2288e7a-9ae9-4e3e-953e-0102cc9bc9bf None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.461318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2288e7a-9ae9-4e3e-953e-0102cc9bc9bf None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:22.461628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2288e7a-9ae9-4e3e-953e-0102cc9bc9bf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.463050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2288e7a-9ae9-4e3e-953e-0102cc9bc9bf None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.469899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 429/859] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:22.469899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea701ab-b35a-4845-8740-35df155b83a1 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.473059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 431/860] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:22 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:22.474039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2288e7a-9ae9-4e3e-953e-0102cc9bc9bf None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.527972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:22.576519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea701ab-b35a-4845-8740-35df155b83a1 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.580529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2288e7a-9ae9-4e3e-953e-0102cc9bc9bf None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.587934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:22.612191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcdcf01-79b3-4f23-bdc1-5f2b0cec72c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.613117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcdcf01-79b3-4f23-bdc1-5f2b0cec72c8 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.613405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcdcf01-79b3-4f23-bdc1-5f2b0cec72c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.613684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcdcf01-79b3-4f23-bdc1-5f2b0cec72c8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.621698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4464b8dc-f021-4f05-a107-79c95f5858cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.621953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcdcf01-79b3-4f23-bdc1-5f2b0cec72c8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.622417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4464b8dc-f021-4f05-a107-79c95f5858cc None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.622703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4464b8dc-f021-4f05-a107-79c95f5858cc None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.622982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4464b8dc-f021-4f05-a107-79c95f5858cc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.623384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcdcf01-79b3-4f23-bdc1-5f2b0cec72c8 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.623901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4464b8dc-f021-4f05-a107-79c95f5858cc None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=eb2c6f06b300473ebafff4503bb24396) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:22.624292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4464b8dc-f021-4f05-a107-79c95f5858cc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.626533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4464b8dc-f021-4f05-a107-79c95f5858cc None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.640073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 432/861] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:22 2022] GET /identity/v3/role_assignments?user.id=eb2c6f06b300473ebafff4503bb24396&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:22.640073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4464b8dc-f021-4f05-a107-79c95f5858cc None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-abcdcf01-79b3-4f23-bdc1-5f2b0cec72c8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:22.640321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:22.641323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:22.642266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 430/862] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:22 2022] GET /identity/v3/users/swift => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:22.652793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcdcf01-79b3-4f23-bdc1-5f2b0cec72c8 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.760356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcdcf01-79b3-4f23-bdc1-5f2b0cec72c8 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.776484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:22.792616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4464b8dc-f021-4f05-a107-79c95f5858cc None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:22.808536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575fcf1f-e3fa-40d3-9904-61d4bafa8acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.815604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575fcf1f-e3fa-40d3-9904-61d4bafa8acc None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.815604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575fcf1f-e3fa-40d3-9904-61d4bafa8acc None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.815604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-575fcf1f-e3fa-40d3-9904-61d4bafa8acc None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.815604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575fcf1f-e3fa-40d3-9904-61d4bafa8acc None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:22.815604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575fcf1f-e3fa-40d3-9904-61d4bafa8acc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.815604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-575fcf1f-e3fa-40d3-9904-61d4bafa8acc None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.841404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 431/863] 10.222.0.25 () {56 vars in 1135 bytes} [Fri Jun 10 12:46:22 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:22.846747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575fcf1f-e3fa-40d3-9904-61d4bafa8acc None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:22.855119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cab41f-e8ef-435a-af55-fe0de0e1fe67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:22.855789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cab41f-e8ef-435a-af55-fe0de0e1fe67 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:22.856072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cab41f-e8ef-435a-af55-fe0de0e1fe67 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:22.856346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cab41f-e8ef-435a-af55-fe0de0e1fe67 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:22.857127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cab41f-e8ef-435a-af55-fe0de0e1fe67 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:22.857436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cab41f-e8ef-435a-af55-fe0de0e1fe67 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:22.858816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60cab41f-e8ef-435a-af55-fe0de0e1fe67 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:22.863706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 433/864] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:22 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:22.971200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-575fcf1f-e3fa-40d3-9904-61d4bafa8acc None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:23.011125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade42b35-db23-4e7c-825f-f297c8bb1dd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:23.011789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade42b35-db23-4e7c-825f-f297c8bb1dd4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:23.012071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade42b35-db23-4e7c-825f-f297c8bb1dd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:23.012350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade42b35-db23-4e7c-825f-f297c8bb1dd4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:23.060032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:23.087386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade42b35-db23-4e7c-825f-f297c8bb1dd4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:23.088871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade42b35-db23-4e7c-825f-f297c8bb1dd4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:23.199505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 432/865] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:22 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:46:23.205200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cab41f-e8ef-435a-af55-fe0de0e1fe67 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:23.245282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:23.315271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cab41f-e8ef-435a-af55-fe0de0e1fe67 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:23.351650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813251f4-9b7d-472d-a94e-cf5e1e69bb7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:23.352442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813251f4-9b7d-472d-a94e-cf5e1e69bb7b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:23.352720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813251f4-9b7d-472d-a94e-cf5e1e69bb7b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:23.353031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813251f4-9b7d-472d-a94e-cf5e1e69bb7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:23.364056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813251f4-9b7d-472d-a94e-cf5e1e69bb7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:23.365551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813251f4-9b7d-472d-a94e-cf5e1e69bb7b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-813251f4-9b7d-472d-a94e-cf5e1e69bb7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:23.383623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:23.384581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:23.385517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:23.385517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:23.385517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:23.385517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:23.385517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 434/866] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:23.393867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade42b35-db23-4e7c-825f-f297c8bb1dd4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:23.465717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:23.501966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade42b35-db23-4e7c-825f-f297c8bb1dd4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:23.541061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d906f2-a643-4f0b-8ddc-e521d42f5878 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:23.542101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d906f2-a643-4f0b-8ddc-e521d42f5878 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:23.542619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d906f2-a643-4f0b-8ddc-e521d42f5878 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:23.543114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d906f2-a643-4f0b-8ddc-e521d42f5878 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:23.544036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d906f2-a643-4f0b-8ddc-e521d42f5878 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:23.544549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d906f2-a643-4f0b-8ddc-e521d42f5878 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:23.546142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d906f2-a643-4f0b-8ddc-e521d42f5878 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:23.564052 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 433/867] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:23.566073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9ad53d-c0df-4c98-a6ea-0c05c96033ed None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:23.566466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9ad53d-c0df-4c98-a6ea-0c05c96033ed None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:23.566845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9ad53d-c0df-4c98-a6ea-0c05c96033ed None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:23.569123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 435/868] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:23.570558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d906f2-a643-4f0b-8ddc-e521d42f5878 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:23.573893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d29d4-0c4d-4130-85e6-9a80bdaffc0a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:23.574181 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d29d4-0c4d-4130-85e6-9a80bdaffc0a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:23.574465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a93d29d4-0c4d-4130-85e6-9a80bdaffc0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:23.592559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a93d29d4-0c4d-4130-85e6-9a80bdaffc0a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:23.594171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a93d29d4-0c4d-4130-85e6-9a80bdaffc0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:23.680289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d906f2-a643-4f0b-8ddc-e521d42f5878 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:23.695155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 436/869] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:23 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:23.715086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:23.717482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93d29d4-0c4d-4130-85e6-9a80bdaffc0a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:23.720661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad648374-a04d-4e89-9b9d-ae1ce5db7a3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:23.721575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad648374-a04d-4e89-9b9d-ae1ce5db7a3c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:23.721860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad648374-a04d-4e89-9b9d-ae1ce5db7a3c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:23.722133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad648374-a04d-4e89-9b9d-ae1ce5db7a3c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:23.729420 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad648374-a04d-4e89-9b9d-ae1ce5db7a3c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:23.731027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad648374-a04d-4e89-9b9d-ae1ce5db7a3c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:23.736907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-ad648374-a04d-4e89-9b9d-ae1ce5db7a3c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:23.746078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:23.746995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:23.747890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:23.747890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:23.747890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:23.747890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:23.747890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:23.747890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:23.747890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:23.747890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:23.747890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 434/870] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:23.754120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad648374-a04d-4e89-9b9d-ae1ce5db7a3c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:23.837962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a93d29d4-0c4d-4130-85e6-9a80bdaffc0a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:23.873272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf2339f-f28f-4e32-8ae2-d87f3b84dd43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:23.874693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf2339f-f28f-4e32-8ae2-d87f3b84dd43 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:23.875206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf2339f-f28f-4e32-8ae2-d87f3b84dd43 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:23.875685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf2339f-f28f-4e32-8ae2-d87f3b84dd43 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:23.876614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf2339f-f28f-4e32-8ae2-d87f3b84dd43 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:23.878207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf2339f-f28f-4e32-8ae2-d87f3b84dd43 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:23.885562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad648374-a04d-4e89-9b9d-ae1ce5db7a3c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:23.920298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb23523-f2eb-410b-aa0a-21b0bd19ae5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:23.921027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb23523-f2eb-410b-aa0a-21b0bd19ae5c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:23.921302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb23523-f2eb-410b-aa0a-21b0bd19ae5c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:23.921724 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb23523-f2eb-410b-aa0a-21b0bd19ae5c None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:23.927040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb23523-f2eb-410b-aa0a-21b0bd19ae5c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:23.927040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb23523-f2eb-410b-aa0a-21b0bd19ae5c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:23.927040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb23523-f2eb-410b-aa0a-21b0bd19ae5c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:23.936732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 435/871] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:23.942223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb23523-f2eb-410b-aa0a-21b0bd19ae5c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:23.958592 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 437/872] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:46:23 2022] POST /identity/v3/services => generated 251 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:46:24.012524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:24.066604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb23523-f2eb-410b-aa0a-21b0bd19ae5c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:24.113606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c078f6-b3b7-4f3a-8062-fce396a5a9d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:24.114278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c078f6-b3b7-4f3a-8062-fce396a5a9d0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:24.114565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c078f6-b3b7-4f3a-8062-fce396a5a9d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:24.114839 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c078f6-b3b7-4f3a-8062-fce396a5a9d0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:24.115717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c078f6-b3b7-4f3a-8062-fce396a5a9d0 None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=bb18cd41c9a14d358aac14ed62ef9e02) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:24.116024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c078f6-b3b7-4f3a-8062-fce396a5a9d0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:24.117619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9c078f6-b3b7-4f3a-8062-fce396a5a9d0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:24.129040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 436/873] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:23 2022] GET /identity/v3/role_assignments?user.id=bb18cd41c9a14d358aac14ed62ef9e02&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=766799d1b6c049168590571f1a959b3b => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:25.044188 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2462b39f-c441-4312-a07c-2aeefcd27e78 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:25.044709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2462b39f-c441-4312-a07c-2aeefcd27e78 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:25.044817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2462b39f-c441-4312-a07c-2aeefcd27e78 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:25.046283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 438/874] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:25.051664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bc4ffb-95b4-44d6-b793-a881f748dcae None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:25.052077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bc4ffb-95b4-44d6-b793-a881f748dcae None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:25.052253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bc4ffb-95b4-44d6-b793-a881f748dcae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:25.071085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-43bc4ffb-95b4-44d6-b793-a881f748dcae None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:25.072308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-43bc4ffb-95b4-44d6-b793-a881f748dcae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:25.148714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 437/875] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:25 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:25.172834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2462b39f-c441-4312-a07c-2aeefcd27e78 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:25.178314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b79417-7051-4554-9a39-fbd310e3c682 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:25.178610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b79417-7051-4554-9a39-fbd310e3c682 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:25.178891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b79417-7051-4554-9a39-fbd310e3c682 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:25.180485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 438/876] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:25.185462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338ace00-f610-409e-a8e4-3f29c317cc7e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:25.185744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338ace00-f610-409e-a8e4-3f29c317cc7e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:25.186018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338ace00-f610-409e-a8e4-3f29c317cc7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:25.204048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-338ace00-f610-409e-a8e4-3f29c317cc7e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:25.205585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-338ace00-f610-409e-a8e4-3f29c317cc7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:25.244687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:25.276241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:25.284842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 439/877] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:25 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:25.307739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338ace00-f610-409e-a8e4-3f29c317cc7e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:25.307992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2462b39f-c441-4312-a07c-2aeefcd27e78 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:25.345296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf8ca70-f4cc-430d-9504-4852be568d28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:25.346309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf8ca70-f4cc-430d-9504-4852be568d28 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:25.346742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf8ca70-f4cc-430d-9504-4852be568d28 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:25.347146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf8ca70-f4cc-430d-9504-4852be568d28 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:25.358528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf8ca70-f4cc-430d-9504-4852be568d28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-bdf8ca70-f4cc-430d-9504-4852be568d28 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:25.366931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:25.368027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:25.369118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:25.378284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 439/878] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Jun 10 12:46:25 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:25.379694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf8ca70-f4cc-430d-9504-4852be568d28 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:25.419156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338ace00-f610-409e-a8e4-3f29c317cc7e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:25.456002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53524b39-fbbc-4cf2-bbb3-26d36e5c3ee3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:25.456669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53524b39-fbbc-4cf2-bbb3-26d36e5c3ee3 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:25.457049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53524b39-fbbc-4cf2-bbb3-26d36e5c3ee3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:25.457412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53524b39-fbbc-4cf2-bbb3-26d36e5c3ee3 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:25.458197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53524b39-fbbc-4cf2-bbb3-26d36e5c3ee3 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:25.459795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53524b39-fbbc-4cf2-bbb3-26d36e5c3ee3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:25.482649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf8ca70-f4cc-430d-9504-4852be568d28 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:25.516891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f3d54a-b622-483a-903f-4b421465105a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:25.517552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f3d54a-b622-483a-903f-4b421465105a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:25.517836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f3d54a-b622-483a-903f-4b421465105a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:25.518114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f3d54a-b622-483a-903f-4b421465105a None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:25.518903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f3d54a-b622-483a-903f-4b421465105a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:25.519215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f3d54a-b622-483a-903f-4b421465105a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:25.520584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f3d54a-b622-483a-903f-4b421465105a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:25.523186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:25.530198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 440/879] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Jun 10 12:46:25 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:25.535504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f3d54a-b622-483a-903f-4b421465105a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:25.568882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 440/880] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:46:25 2022] POST /identity/v3/services => generated 269 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:46:25.655055 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f3d54a-b622-483a-903f-4b421465105a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:25.699654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b130a2b6-2981-4368-b71a-302e4169dcb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:25.700453 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b130a2b6-2981-4368-b71a-302e4169dcb0 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:25.700736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b130a2b6-2981-4368-b71a-302e4169dcb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:25.701045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b130a2b6-2981-4368-b71a-302e4169dcb0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:25.712723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b130a2b6-2981-4368-b71a-302e4169dcb0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:25.714240 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b130a2b6-2981-4368-b71a-302e4169dcb0 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-b130a2b6-2981-4368-b71a-302e4169dcb0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:25.733299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:25.734260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:25.735139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:25.735139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:25.735139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:25.735139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:25.739231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 441/881] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:25.741186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53524b39-fbbc-4cf2-bbb3-26d36e5c3ee3 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:25.776122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:25.848599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53524b39-fbbc-4cf2-bbb3-26d36e5c3ee3 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:25.889668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c8c989-7f0d-4847-8f1e-30a09417b91c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:25.890255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c8c989-7f0d-4847-8f1e-30a09417b91c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:25.890542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c8c989-7f0d-4847-8f1e-30a09417b91c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:25.890820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c8c989-7f0d-4847-8f1e-30a09417b91c None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:25.891680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c8c989-7f0d-4847-8f1e-30a09417b91c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:25.891993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c8c989-7f0d-4847-8f1e-30a09417b91c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:25.893426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c8c989-7f0d-4847-8f1e-30a09417b91c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:25.911531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 441/882] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:25.912961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b130a2b6-2981-4368-b71a-302e4169dcb0 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:25.963235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:26.019077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b130a2b6-2981-4368-b71a-302e4169dcb0 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:26.057100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d158923-c6a7-46a5-a307-246bd8163ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:26.058688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d158923-c6a7-46a5-a307-246bd8163ff9 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.059203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d158923-c6a7-46a5-a307-246bd8163ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.059685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d158923-c6a7-46a5-a307-246bd8163ff9 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.067209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d158923-c6a7-46a5-a307-246bd8163ff9 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:26.068946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d158923-c6a7-46a5-a307-246bd8163ff9 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-9d158923-c6a7-46a5-a307-246bd8163ff9 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:26.084504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:26.085495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:26.086445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:26.089276 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 442/883] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:46:25 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:26.095050 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c8c989-7f0d-4847-8f1e-30a09417b91c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:26.167132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e868062-1317-4f87-8085-a452ae9b9dd3 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.167439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e868062-1317-4f87-8085-a452ae9b9dd3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.167606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e868062-1317-4f87-8085-a452ae9b9dd3 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.169260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 443/884] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:26.175216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d49ae4d-6501-43cd-a48e-8175cc498738 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.175494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d49ae4d-6501-43cd-a48e-8175cc498738 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.175775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d49ae4d-6501-43cd-a48e-8175cc498738 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.196441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-4d49ae4d-6501-43cd-a48e-8175cc498738 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:26.198491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4d49ae4d-6501-43cd-a48e-8175cc498738 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:26.214924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:26.225937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c8c989-7f0d-4847-8f1e-30a09417b91c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:26.268477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96c77df-7ba6-4800-8077-91af430267ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:26.269587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96c77df-7ba6-4800-8077-91af430267ae None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.270094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96c77df-7ba6-4800-8077-91af430267ae None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.270597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96c77df-7ba6-4800-8077-91af430267ae None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.272371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96c77df-7ba6-4800-8077-91af430267ae None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:26.272919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96c77df-7ba6-4800-8077-91af430267ae None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:26.274682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96c77df-7ba6-4800-8077-91af430267ae None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:26.285491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 444/885] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:26 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:26.300800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 442/886] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Jun 10 12:46:26 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:26.305823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d49ae4d-6501-43cd-a48e-8175cc498738 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:26.306259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96c77df-7ba6-4800-8077-91af430267ae None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:26.349842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:26.412175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96c77df-7ba6-4800-8077-91af430267ae None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:26.413026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d49ae4d-6501-43cd-a48e-8175cc498738 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:26.446480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9137dc8-c767-4a86-91bc-1ad507e86727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:26.447119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9137dc8-c767-4a86-91bc-1ad507e86727 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.447402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9137dc8-c767-4a86-91bc-1ad507e86727 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.447705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9137dc8-c767-4a86-91bc-1ad507e86727 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.449033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829b7bdb-bc2b-494a-94e6-b789d56991db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:26.449980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829b7bdb-bc2b-494a-94e6-b789d56991db None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.450501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829b7bdb-bc2b-494a-94e6-b789d56991db None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.451041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829b7bdb-bc2b-494a-94e6-b789d56991db None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.451704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829b7bdb-bc2b-494a-94e6-b789d56991db None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:26.453096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829b7bdb-bc2b-494a-94e6-b789d56991db None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-829b7bdb-bc2b-494a-94e6-b789d56991db None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:26.466449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:26.467660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:26.468499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 443/887] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:46:26 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:26.469466 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ca7f61-c8a4-4b7e-b41a-e648a5efbd3b None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.469534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ca7f61-c8a4-4b7e-b41a-e648a5efbd3b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.469776 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ca7f61-c8a4-4b7e-b41a-e648a5efbd3b None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.474557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 444/888] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:26.474557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ca7f61-c8a4-4b7e-b41a-e648a5efbd3b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:26.490949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:26.527538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9137dc8-c767-4a86-91bc-1ad507e86727 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:26.529023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9137dc8-c767-4a86-91bc-1ad507e86727 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:26.562742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:26.588848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ca7f61-c8a4-4b7e-b41a-e648a5efbd3b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:26.625091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1a5d80-56c1-4357-a609-eceb9d9bfe5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:26.625749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1a5d80-56c1-4357-a609-eceb9d9bfe5e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.626031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1a5d80-56c1-4357-a609-eceb9d9bfe5e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.626309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1a5d80-56c1-4357-a609-eceb9d9bfe5e None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.627006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1a5d80-56c1-4357-a609-eceb9d9bfe5e None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:26.627318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1a5d80-56c1-4357-a609-eceb9d9bfe5e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:26.633051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 445/889] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:26 2022] GET /identity/v3/auth/tokens => generated 1362 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:46:26.633206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1a5d80-56c1-4357-a609-eceb9d9bfe5e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:26.633311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff49ff4-bd96-4fe4-aa22-89ae021e0892 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.633311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff49ff4-bd96-4fe4-aa22-89ae021e0892 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.633311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff49ff4-bd96-4fe4-aa22-89ae021e0892 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.638300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 445/890] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:46:26 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:26.642854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1a5d80-56c1-4357-a609-eceb9d9bfe5e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:26.650492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1ff49ff4-bd96-4fe4-aa22-89ae021e0892 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:26.651675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1ff49ff4-bd96-4fe4-aa22-89ae021e0892 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:26.725237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 446/891] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:26 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:26.725237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff49ff4-bd96-4fe4-aa22-89ae021e0892 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:26.726778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:26.750412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1a5d80-56c1-4357-a609-eceb9d9bfe5e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:26.784779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bf218b-c9e5-468b-9b6b-3d1ef6addde6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:26.785582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bf218b-c9e5-468b-9b6b-3d1ef6addde6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.785866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bf218b-c9e5-468b-9b6b-3d1ef6addde6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.786137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11bf218b-c9e5-468b-9b6b-3d1ef6addde6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.796140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bf218b-c9e5-468b-9b6b-3d1ef6addde6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:26.797642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11bf218b-c9e5-468b-9b6b-3d1ef6addde6 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:26.810267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-11bf218b-c9e5-468b-9b6b-3d1ef6addde6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:26.817107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:26.818141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:26.819071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:26.819071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:26.819071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:26.819071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:26.819071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 446/892] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:26.819893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bf218b-c9e5-468b-9b6b-3d1ef6addde6 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:26.852739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff49ff4-bd96-4fe4-aa22-89ae021e0892 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:26.886733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7abbd24-d9bb-405d-9448-57ddb70a71a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:26.887400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7abbd24-d9bb-405d-9448-57ddb70a71a3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.887678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7abbd24-d9bb-405d-9448-57ddb70a71a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.887949 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7abbd24-d9bb-405d-9448-57ddb70a71a3 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.888653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7abbd24-d9bb-405d-9448-57ddb70a71a3 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:26.889016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7abbd24-d9bb-405d-9448-57ddb70a71a3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:26.890388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7abbd24-d9bb-405d-9448-57ddb70a71a3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:26.900655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 447/893] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:46:26 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:26.900655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7abbd24-d9bb-405d-9448-57ddb70a71a3 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:26.941947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11bf218b-c9e5-468b-9b6b-3d1ef6addde6 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:26.972473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:26.978958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbbea14-9506-48d6-b611-5c0cf00ec056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:26.979789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbbea14-9506-48d6-b611-5c0cf00ec056 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:26.980219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbbea14-9506-48d6-b611-5c0cf00ec056 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:26.980567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbbea14-9506-48d6-b611-5c0cf00ec056 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:26.981344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbbea14-9506-48d6-b611-5c0cf00ec056 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:26.982787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbbea14-9506-48d6-b611-5c0cf00ec056 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-bbbbea14-9506-48d6-b611-5c0cf00ec056 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:26.998032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:26.999217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:27.000255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:27.004426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 447/894] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:26 2022] GET /identity/v3/services/object-store => generated 93 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:27.005699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbbea14-9506-48d6-b611-5c0cf00ec056 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:27.008064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7abbd24-d9bb-405d-9448-57ddb70a71a3 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:27.039196 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:27.045734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4b4a0f-c353-4103-89eb-fb09919585c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:27.046389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b4a0f-c353-4103-89eb-fb09919585c6 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.046670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b4a0f-c353-4103-89eb-fb09919585c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.047406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd4b4a0f-c353-4103-89eb-fb09919585c6 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.048114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4b4a0f-c353-4103-89eb-fb09919585c6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:27.048420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4b4a0f-c353-4103-89eb-fb09919585c6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:27.050460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd4b4a0f-c353-4103-89eb-fb09919585c6 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:27.062589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 448/895] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:27.065943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4b4a0f-c353-4103-89eb-fb09919585c6 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:27.110246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbbea14-9506-48d6-b611-5c0cf00ec056 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:27.144887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b69f5de-a901-4a38-b6e0-55b800ba8674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:27.145657 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69f5de-a901-4a38-b6e0-55b800ba8674 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.145941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69f5de-a901-4a38-b6e0-55b800ba8674 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.146233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69f5de-a901-4a38-b6e0-55b800ba8674 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.147169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b69f5de-a901-4a38-b6e0-55b800ba8674 None admin] RBAC: Adding query filter params (interface=public, service_id=9e7c98afb33b4336b039c3481f355b60, region_id=RegionOne) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:27.147485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b69f5de-a901-4a38-b6e0-55b800ba8674 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:27.148882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b69f5de-a901-4a38-b6e0-55b800ba8674 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:27.161354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 448/896] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Jun 10 12:46:27 2022] GET /identity/v3/endpoints?service_id=9e7c98afb33b4336b039c3481f355b60&interface=public®ion_id=RegionOne => generated 194 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:27.161354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b69f5de-a901-4a38-b6e0-55b800ba8674 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:27.173075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd4b4a0f-c353-4103-89eb-fb09919585c6 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:27.206441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214d9f6e-4deb-4e9a-9754-76e6c91d9777 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:27.207099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214d9f6e-4deb-4e9a-9754-76e6c91d9777 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.207383 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214d9f6e-4deb-4e9a-9754-76e6c91d9777 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.207664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214d9f6e-4deb-4e9a-9754-76e6c91d9777 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.208362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214d9f6e-4deb-4e9a-9754-76e6c91d9777 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:27.208670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214d9f6e-4deb-4e9a-9754-76e6c91d9777 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:27.210118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214d9f6e-4deb-4e9a-9754-76e6c91d9777 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:27.214618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 449/897] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:27 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:27.216968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214d9f6e-4deb-4e9a-9754-76e6c91d9777 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:27.224826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:27.269051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b69f5de-a901-4a38-b6e0-55b800ba8674 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:27.276012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:27.304727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d9982b-c743-4e05-a691-d10cb5915866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:27.305733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9982b-c743-4e05-a691-d10cb5915866 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.305843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9982b-c743-4e05-a691-d10cb5915866 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.306108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d9982b-c743-4e05-a691-d10cb5915866 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.312909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d9982b-c743-4e05-a691-d10cb5915866 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:27.314357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d9982b-c743-4e05-a691-d10cb5915866 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:27.326888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214d9f6e-4deb-4e9a-9754-76e6c91d9777 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-52d9982b-c743-4e05-a691-d10cb5915866 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:27.329685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:27.330578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:27.331754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:27.331754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:27.331754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:27.331754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:27.331754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:27.331754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:27.331754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:27.331754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:27.331754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 449/898] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:27.332407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d9982b-c743-4e05-a691-d10cb5915866 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:27.370531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc856541-a947-4308-ae30-12f4c3f14e6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:27.371288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc856541-a947-4308-ae30-12f4c3f14e6e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.371575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc856541-a947-4308-ae30-12f4c3f14e6e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.371850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc856541-a947-4308-ae30-12f4c3f14e6e None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.372985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc856541-a947-4308-ae30-12f4c3f14e6e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:27.374713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc856541-a947-4308-ae30-12f4c3f14e6e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:27.392797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 450/899] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Jun 10 12:46:27 2022] GET /identity/v3/services => generated 1268 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 12:46:27.392797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc856541-a947-4308-ae30-12f4c3f14e6e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:27.447710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d9982b-c743-4e05-a691-d10cb5915866 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:27.468460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:27.489583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc120a6-b022-43fe-884f-6a1e1554eddf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:27.490902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc120a6-b022-43fe-884f-6a1e1554eddf None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.491251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc120a6-b022-43fe-884f-6a1e1554eddf None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.491618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc120a6-b022-43fe-884f-6a1e1554eddf None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.492412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc120a6-b022-43fe-884f-6a1e1554eddf None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:27.492806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc120a6-b022-43fe-884f-6a1e1554eddf None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:27.494427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc120a6-b022-43fe-884f-6a1e1554eddf None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:27.501160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 450/900] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:27 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:27.519249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc856541-a947-4308-ae30-12f4c3f14e6e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:27.541828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:27.561481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8945885-01e1-4cf9-90d4-2c31f3fb0dcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:27.562163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8945885-01e1-4cf9-90d4-2c31f3fb0dcd None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.562582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8945885-01e1-4cf9-90d4-2c31f3fb0dcd None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.562865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8945885-01e1-4cf9-90d4-2c31f3fb0dcd None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.563730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8945885-01e1-4cf9-90d4-2c31f3fb0dcd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:27.567630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8945885-01e1-4cf9-90d4-2c31f3fb0dcd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:27.567630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8945885-01e1-4cf9-90d4-2c31f3fb0dcd None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:27.581294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 451/901] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:27.588061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc120a6-b022-43fe-884f-6a1e1554eddf None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:27.705438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc120a6-b022-43fe-884f-6a1e1554eddf None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:27.740892 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4fdfd5-a8de-409b-8318-270a105bf4aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:27.741548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4fdfd5-a8de-409b-8318-270a105bf4aa None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.741832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4fdfd5-a8de-409b-8318-270a105bf4aa None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.742108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4fdfd5-a8de-409b-8318-270a105bf4aa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.742941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4fdfd5-a8de-409b-8318-270a105bf4aa None admin] RBAC: Adding query filter params (role.id=ee9ee7e94d4e4ac39a965a63e0abea25, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=eb2c6f06b300473ebafff4503bb24396) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:27.743253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4fdfd5-a8de-409b-8318-270a105bf4aa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:27.744692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4fdfd5-a8de-409b-8318-270a105bf4aa None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:27.750899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 451/902] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:27 2022] GET /identity/v3/role_assignments?user.id=eb2c6f06b300473ebafff4503bb24396&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=ee9ee7e94d4e4ac39a965a63e0abea25 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:27.858024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6048fa56-9bfb-47a6-aadd-c0fd88d53dd4 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.858225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6048fa56-9bfb-47a6-aadd-c0fd88d53dd4 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.858489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6048fa56-9bfb-47a6-aadd-c0fd88d53dd4 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.860158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 452/903] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:27.865651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210c960e-5221-4e65-b54a-d57431c39762 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:27.865941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210c960e-5221-4e65-b54a-d57431c39762 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:27.866224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210c960e-5221-4e65-b54a-d57431c39762 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:27.885067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-210c960e-5221-4e65-b54a-d57431c39762 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:27.886287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-210c960e-5221-4e65-b54a-d57431c39762 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:27.936828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:27.962030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 452/904] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:27 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:27.985195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6048fa56-9bfb-47a6-aadd-c0fd88d53dd4 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:28.092489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6048fa56-9bfb-47a6-aadd-c0fd88d53dd4 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:28.128376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3d0faf-eccf-4a66-a922-c9c676c7a4ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:28.129192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d0faf-eccf-4a66-a922-c9c676c7a4ce None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:28.129487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d0faf-eccf-4a66-a922-c9c676c7a4ce None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:28.129792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3d0faf-eccf-4a66-a922-c9c676c7a4ce None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:28.130445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3d0faf-eccf-4a66-a922-c9c676c7a4ce None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:28.131884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3d0faf-eccf-4a66-a922-c9c676c7a4ce None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-cb3d0faf-eccf-4a66-a922-c9c676c7a4ce None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:28.144462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:28.145419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:28.146783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 453/905] 10.222.0.25 () {56 vars in 1132 bytes} [Fri Jun 10 12:46:27 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:28.154328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210c960e-5221-4e65-b54a-d57431c39762 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:28.263304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210c960e-5221-4e65-b54a-d57431c39762 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:28.299728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dace6b-3985-4eb6-86a9-5ee145ccfe5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:28.300390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dace6b-3985-4eb6-86a9-5ee145ccfe5b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:28.300674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dace6b-3985-4eb6-86a9-5ee145ccfe5b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:28.300982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dace6b-3985-4eb6-86a9-5ee145ccfe5b None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:28.301690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dace6b-3985-4eb6-86a9-5ee145ccfe5b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:28.301998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dace6b-3985-4eb6-86a9-5ee145ccfe5b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:28.303910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dace6b-3985-4eb6-86a9-5ee145ccfe5b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:28.309044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 453/906] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:28 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:28.314423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3d0faf-eccf-4a66-a922-c9c676c7a4ce None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:28.359524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:28.416880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3d0faf-eccf-4a66-a922-c9c676c7a4ce None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:28.451538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52d2bdf-6c7d-475b-afcc-0dc28916277b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:28.452201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d2bdf-6c7d-475b-afcc-0dc28916277b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:28.452481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d2bdf-6c7d-475b-afcc-0dc28916277b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:28.452754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52d2bdf-6c7d-475b-afcc-0dc28916277b None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:28.453475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52d2bdf-6c7d-475b-afcc-0dc28916277b None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:28.453779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52d2bdf-6c7d-475b-afcc-0dc28916277b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:28.455196 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52d2bdf-6c7d-475b-afcc-0dc28916277b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:28.460657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 454/907] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:28 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:28.466362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dace6b-3985-4eb6-86a9-5ee145ccfe5b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:28.510841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:28.571362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dace6b-3985-4eb6-86a9-5ee145ccfe5b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:28.608190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d304731-dcc6-4c63-8c57-49c8a0e8fc28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:28.608873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d304731-dcc6-4c63-8c57-49c8a0e8fc28 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:28.609156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d304731-dcc6-4c63-8c57-49c8a0e8fc28 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:28.609429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d304731-dcc6-4c63-8c57-49c8a0e8fc28 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:28.610211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d304731-dcc6-4c63-8c57-49c8a0e8fc28 None admin] RBAC: Adding query filter params (interface=public, service_id=0f03bb01a1d8456d9b34970bebcd2d30, region_id=RegionOne) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:28.610517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d304731-dcc6-4c63-8c57-49c8a0e8fc28 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:28.611877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d304731-dcc6-4c63-8c57-49c8a0e8fc28 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:28.618073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 454/908] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Jun 10 12:46:28 2022] GET /identity/v3/endpoints?service_id=0f03bb01a1d8456d9b34970bebcd2d30&interface=public®ion_id=RegionOne => generated 194 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:28.623599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52d2bdf-6c7d-475b-afcc-0dc28916277b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:28.727793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52d2bdf-6c7d-475b-afcc-0dc28916277b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:28.757233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:28.761176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdbd6e7c-0da4-4a47-8f5a-d285beffac3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:28.761837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbd6e7c-0da4-4a47-8f5a-d285beffac3b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:28.762118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbd6e7c-0da4-4a47-8f5a-d285beffac3b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:28.762394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbd6e7c-0da4-4a47-8f5a-d285beffac3b None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:28.763068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbd6e7c-0da4-4a47-8f5a-d285beffac3b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:28.764433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdbd6e7c-0da4-4a47-8f5a-d285beffac3b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:28.770501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 455/909] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Jun 10 12:46:28 2022] GET /identity/v3/services => generated 1268 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 12:46:29.629296 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aa0591-7426-442b-ad51-28549519b731 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:29.629945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aa0591-7426-442b-ad51-28549519b731 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:29.629945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9aa0591-7426-442b-ad51-28549519b731 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:29.631574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 455/910] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:29.636862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b26c894-93b4-4ddf-b771-e303c08c2749 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:29.637286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b26c894-93b4-4ddf-b771-e303c08c2749 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:29.637464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b26c894-93b4-4ddf-b771-e303c08c2749 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:29.656363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-6b26c894-93b4-4ddf-b771-e303c08c2749 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:29.657652 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6b26c894-93b4-4ddf-b771-e303c08c2749 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:29.731192 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f2c38-ba0d-4c94-b263-7311cdd3a426 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:29.731833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f2c38-ba0d-4c94-b263-7311cdd3a426 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:29.732326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b72f2c38-ba0d-4c94-b263-7311cdd3a426 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:29.732705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 456/911] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:29 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:29.734123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 456/912] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:29.739448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a27e4e-cc29-4ec8-ad1d-304256458eec None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:29.739989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a27e4e-cc29-4ec8-ad1d-304256458eec None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:29.740467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a27e4e-cc29-4ec8-ad1d-304256458eec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:29.755351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f2c38-ba0d-4c94-b263-7311cdd3a426 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:29.757234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-45a27e4e-cc29-4ec8-ad1d-304256458eec None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:29.758621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-45a27e4e-cc29-4ec8-ad1d-304256458eec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:29.849120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 457/913] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:29 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:29.871748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a27e4e-cc29-4ec8-ad1d-304256458eec None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:29.873867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b72f2c38-ba0d-4c94-b263-7311cdd3a426 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:29.877472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:29.912514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:29.927863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b2274f-1a9d-4fd8-9338-89708e9cd314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:29.929659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b2274f-1a9d-4fd8-9338-89708e9cd314 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:29.930389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b2274f-1a9d-4fd8-9338-89708e9cd314 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:29.930918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b2274f-1a9d-4fd8-9338-89708e9cd314 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:29.931948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b2274f-1a9d-4fd8-9338-89708e9cd314 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:29.934368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b2274f-1a9d-4fd8-9338-89708e9cd314 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-a4b2274f-1a9d-4fd8-9338-89708e9cd314 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:29.951112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:29.952393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jun 10 12:46:29.953328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:29.961138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 457/914] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:46:29 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:29.965921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b2274f-1a9d-4fd8-9338-89708e9cd314 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:30.000248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a27e4e-cc29-4ec8-ad1d-304256458eec None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:30.039833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17626b69-e621-4550-b30e-0c4e401316c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:30.040508 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17626b69-e621-4550-b30e-0c4e401316c9 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.040814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17626b69-e621-4550-b30e-0c4e401316c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.041103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17626b69-e621-4550-b30e-0c4e401316c9 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.041840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17626b69-e621-4550-b30e-0c4e401316c9 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:30.043211 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17626b69-e621-4550-b30e-0c4e401316c9 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:30.070427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b2274f-1a9d-4fd8-9338-89708e9cd314 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:30.103040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 458/915] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:46:29 2022] POST /identity/v3/services => generated 241 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:46:30.104150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db00c85-2f88-4670-84e9-84d478051263 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.104437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db00c85-2f88-4670-84e9-84d478051263 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.104709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db00c85-2f88-4670-84e9-84d478051263 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.106893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 459/916] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:30.111723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3808e2-b404-4c5f-a743-c97026fa571c None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.112007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3808e2-b404-4c5f-a743-c97026fa571c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.112335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3808e2-b404-4c5f-a743-c97026fa571c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.122108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8775c555-2881-41e2-937a-9a45c877d52c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:30.123187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8775c555-2881-41e2-937a-9a45c877d52c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.123679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8775c555-2881-41e2-937a-9a45c877d52c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.124223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8775c555-2881-41e2-937a-9a45c877d52c None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.125913 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8775c555-2881-41e2-937a-9a45c877d52c None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:30.126523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8775c555-2881-41e2-937a-9a45c877d52c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:30.128576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8775c555-2881-41e2-937a-9a45c877d52c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:30.131526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9c3808e2-b404-4c5f-a743-c97026fa571c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:30.132673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9c3808e2-b404-4c5f-a743-c97026fa571c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:30.136031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 458/917] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:46:29 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:30.142515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8775c555-2881-41e2-937a-9a45c877d52c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:30.184597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:30.230463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 460/918] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:30 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:30.253276 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3808e2-b404-4c5f-a743-c97026fa571c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:30.274751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8775c555-2881-41e2-937a-9a45c877d52c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:30.278617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:30.325879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f8f999-a9c5-491f-a4bd-fedcc374e66c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:30.326879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f8f999-a9c5-491f-a4bd-fedcc374e66c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.327312 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f8f999-a9c5-491f-a4bd-fedcc374e66c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.327740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f8f999-a9c5-491f-a4bd-fedcc374e66c None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.328578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f8f999-a9c5-491f-a4bd-fedcc374e66c None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:30.329042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f8f999-a9c5-491f-a4bd-fedcc374e66c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:30.330639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f8f999-a9c5-491f-a4bd-fedcc374e66c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:30.339669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 459/919] 10.222.0.25 () {56 vars in 1114 bytes} [Fri Jun 10 12:46:30 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:30.346670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f8f999-a9c5-491f-a4bd-fedcc374e66c None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:30.385776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3808e2-b404-4c5f-a743-c97026fa571c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:30.433034 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1493ce05-a1aa-4d2e-b5c2-3aa21918e379 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:30.434119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1493ce05-a1aa-4d2e-b5c2-3aa21918e379 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.435323 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1493ce05-a1aa-4d2e-b5c2-3aa21918e379 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.435560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1493ce05-a1aa-4d2e-b5c2-3aa21918e379 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.450239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1493ce05-a1aa-4d2e-b5c2-3aa21918e379 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:30.461840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-1493ce05-a1aa-4d2e-b5c2-3aa21918e379 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:30.461984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:30.462934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:30.463796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:30.464450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 461/920] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Jun 10 12:46:30 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:30.471955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1493ce05-a1aa-4d2e-b5c2-3aa21918e379 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:30.476419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f8f999-a9c5-491f-a4bd-fedcc374e66c None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:30.513076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684dd255-fff1-41d9-92b5-f2347a6725f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:30.513787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684dd255-fff1-41d9-92b5-f2347a6725f2 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.514070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684dd255-fff1-41d9-92b5-f2347a6725f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.514389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684dd255-fff1-41d9-92b5-f2347a6725f2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.515131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684dd255-fff1-41d9-92b5-f2347a6725f2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:30.516505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684dd255-fff1-41d9-92b5-f2347a6725f2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:30.570078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:30.589207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1493ce05-a1aa-4d2e-b5c2-3aa21918e379 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:30.596348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 460/921] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Jun 10 12:46:30 2022] POST /identity/v3/endpoints => generated 346 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:46:30.645593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c078754-a71e-45d6-bdf9-2c7f7cafb965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:30.646352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c078754-a71e-45d6-bdf9-2c7f7cafb965 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.646640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c078754-a71e-45d6-bdf9-2c7f7cafb965 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.646918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c078754-a71e-45d6-bdf9-2c7f7cafb965 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.647680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c078754-a71e-45d6-bdf9-2c7f7cafb965 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:30.647993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c078754-a71e-45d6-bdf9-2c7f7cafb965 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:30.649451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c078754-a71e-45d6-bdf9-2c7f7cafb965 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:30.666406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 462/922] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Jun 10 12:46:30 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:30.672072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684dd255-fff1-41d9-92b5-f2347a6725f2 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:30.785858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684dd255-fff1-41d9-92b5-f2347a6725f2 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:30.822190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8f7f71-944f-4501-bf3b-b2c9b94046d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:30.822993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8f7f71-944f-4501-bf3b-b2c9b94046d1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.823275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8f7f71-944f-4501-bf3b-b2c9b94046d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.823552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8f7f71-944f-4501-bf3b-b2c9b94046d1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.835056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8f7f71-944f-4501-bf3b-b2c9b94046d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:30.836530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8f7f71-944f-4501-bf3b-b2c9b94046d1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:30.845647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-6a8f7f71-944f-4501-bf3b-b2c9b94046d1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:30.855506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:30.873942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:30.875001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:30.875001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:30.875001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:30.875001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:30.875001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 461/923] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:30.875307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c078754-a71e-45d6-bdf9-2c7f7cafb965 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:30.982512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dc40d-2d62-418f-a98b-699faece5263 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.982875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dc40d-2d62-418f-a98b-699faece5263 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.983072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2dc40d-2d62-418f-a98b-699faece5263 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:30.984438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 462/924] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:30.984638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c078754-a71e-45d6-bdf9-2c7f7cafb965 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:30.990939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6fd53b-8908-4b24-a5bd-6adba6a8c0f1 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:30.991533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6fd53b-8908-4b24-a5bd-6adba6a8c0f1 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:30.992012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6fd53b-8908-4b24-a5bd-6adba6a8c0f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.009515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ea6fd53b-8908-4b24-a5bd-6adba6a8c0f1 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:31.011052 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ea6fd53b-8908-4b24-a5bd-6adba6a8c0f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:31.030205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729bb6d4-bbbd-4d1f-83c7-8870f7b3829f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:31.030877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729bb6d4-bbbd-4d1f-83c7-8870f7b3829f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:31.031163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729bb6d4-bbbd-4d1f-83c7-8870f7b3829f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:31.031444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-729bb6d4-bbbd-4d1f-83c7-8870f7b3829f None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.032573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729bb6d4-bbbd-4d1f-83c7-8870f7b3829f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:31.032937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729bb6d4-bbbd-4d1f-83c7-8870f7b3829f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:31.034915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-729bb6d4-bbbd-4d1f-83c7-8870f7b3829f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:31.045569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:31.057068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 463/925] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:31.058699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729bb6d4-bbbd-4d1f-83c7-8870f7b3829f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:31.099795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 463/926] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:30 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:31.123638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6fd53b-8908-4b24-a5bd-6adba6a8c0f1 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:31.191199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-729bb6d4-bbbd-4d1f-83c7-8870f7b3829f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:31.231488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6fd53b-8908-4b24-a5bd-6adba6a8c0f1 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:31.237595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36f63be-ec5e-4c69-bcfc-7a4d9318a207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:31.238705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f63be-ec5e-4c69-bcfc-7a4d9318a207 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:31.239215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f63be-ec5e-4c69-bcfc-7a4d9318a207 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:31.240654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36f63be-ec5e-4c69-bcfc-7a4d9318a207 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.251355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36f63be-ec5e-4c69-bcfc-7a4d9318a207 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:31.253100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36f63be-ec5e-4c69-bcfc-7a4d9318a207 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:31.263588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:31.270284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5731a0b-701d-4935-bf61-bdcb83f8aad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:31.274130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5731a0b-701d-4935-bf61-bdcb83f8aad4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:31.274130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5731a0b-701d-4935-bf61-bdcb83f8aad4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:31.274130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5731a0b-701d-4935-bf61-bdcb83f8aad4 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.274130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5731a0b-701d-4935-bf61-bdcb83f8aad4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:31.274476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5731a0b-701d-4935-bf61-bdcb83f8aad4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-a36f63be-ec5e-4c69-bcfc-7a4d9318a207 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:31.277873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:31.278866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:31.279813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:31.286758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 464/927] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:46:31 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:31.288130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36f63be-ec5e-4c69-bcfc-7a4d9318a207 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-d5731a0b-701d-4935-bf61-bdcb83f8aad4 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:31.291928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:31.293098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jun 10 12:46:31.293948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:31.299279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 464/928] 10.222.0.25 () {56 vars in 1132 bytes} [Fri Jun 10 12:46:31 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:31.300143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5731a0b-701d-4935-bf61-bdcb83f8aad4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:31.363652 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:31.411821 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36f63be-ec5e-4c69-bcfc-7a4d9318a207 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:31.434746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5731a0b-701d-4935-bf61-bdcb83f8aad4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:31.447426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f978d63-8f91-44d1-9f28-930382011887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:31.448172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f978d63-8f91-44d1-9f28-930382011887 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:31.448522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f978d63-8f91-44d1-9f28-930382011887 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:31.448913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f978d63-8f91-44d1-9f28-930382011887 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.449817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f978d63-8f91-44d1-9f28-930382011887 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:31.450188 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f978d63-8f91-44d1-9f28-930382011887 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:31.451697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f978d63-8f91-44d1-9f28-930382011887 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:31.469552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 465/929] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Jun 10 12:46:31 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:31.470523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d897ca-3d7f-4746-baed-f2bb2270602e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:31.471190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d897ca-3d7f-4746-baed-f2bb2270602e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:31.471480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d897ca-3d7f-4746-baed-f2bb2270602e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:31.471761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d897ca-3d7f-4746-baed-f2bb2270602e None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.472471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d897ca-3d7f-4746-baed-f2bb2270602e None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:31.472797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d897ca-3d7f-4746-baed-f2bb2270602e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:31.474830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d897ca-3d7f-4746-baed-f2bb2270602e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:31.476913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f978d63-8f91-44d1-9f28-930382011887 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:31.480316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 465/930] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:31 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:31.485779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d897ca-3d7f-4746-baed-f2bb2270602e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:31.547202 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:31.577827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f978d63-8f91-44d1-9f28-930382011887 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:31.592305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d897ca-3d7f-4746-baed-f2bb2270602e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:31.609126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:31.617353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac28e67-a2e1-4e04-865f-c483eab1018c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:31.618397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac28e67-a2e1-4e04-865f-c483eab1018c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:31.618881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac28e67-a2e1-4e04-865f-c483eab1018c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:31.619357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac28e67-a2e1-4e04-865f-c483eab1018c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.627537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d5d4c-4758-4fe5-a1f4-efb0f2774b77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:31.628198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d5d4c-4758-4fe5-a1f4-efb0f2774b77 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:31.628483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d5d4c-4758-4fe5-a1f4-efb0f2774b77 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:31.628877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74d5d4c-4758-4fe5-a1f4-efb0f2774b77 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.629590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74d5d4c-4758-4fe5-a1f4-efb0f2774b77 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:31.629901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74d5d4c-4758-4fe5-a1f4-efb0f2774b77 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:31.631822 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74d5d4c-4758-4fe5-a1f4-efb0f2774b77 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:31.632551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac28e67-a2e1-4e04-865f-c483eab1018c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:31.634365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac28e67-a2e1-4e04-865f-c483eab1018c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:31.641198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 466/931] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:31 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:31.642974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d5d4c-4758-4fe5-a1f4-efb0f2774b77 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-5ac28e67-a2e1-4e04-865f-c483eab1018c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:31.654429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:31.655653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:31.656552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:31.656552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:31.656552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:31.656552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:31.663013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 466/932] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:31.664586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac28e67-a2e1-4e04-865f-c483eab1018c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:31.758238 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74d5d4c-4758-4fe5-a1f4-efb0f2774b77 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:31.783205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:31.795523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac28e67-a2e1-4e04-865f-c483eab1018c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:31.796530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03736c10-95fa-4ce1-85a7-518ed86a6fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:31.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03736c10-95fa-4ce1-85a7-518ed86a6fe8 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:31.797519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03736c10-95fa-4ce1-85a7-518ed86a6fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:31.797845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03736c10-95fa-4ce1-85a7-518ed86a6fe8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.798600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03736c10-95fa-4ce1-85a7-518ed86a6fe8 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:31.800215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03736c10-95fa-4ce1-85a7-518ed86a6fe8 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:31.834611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4a8f6d-0ed2-4a4d-8b57-7b78f44f13e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:31.835429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4a8f6d-0ed2-4a4d-8b57-7b78f44f13e7 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:31.835789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4a8f6d-0ed2-4a4d-8b57-7b78f44f13e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:31.836138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4a8f6d-0ed2-4a4d-8b57-7b78f44f13e7 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:31.837032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4a8f6d-0ed2-4a4d-8b57-7b78f44f13e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:31.837414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4a8f6d-0ed2-4a4d-8b57-7b78f44f13e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:31.838892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4a8f6d-0ed2-4a4d-8b57-7b78f44f13e7 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:31.856830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 467/934] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:31.857043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 467/934] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Jun 10 12:46:31 2022] POST /identity/v3/endpoints => generated 347 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:46:31.857356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4a8f6d-0ed2-4a4d-8b57-7b78f44f13e7 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:31.892223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:31.971901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4a8f6d-0ed2-4a4d-8b57-7b78f44f13e7 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:32.015658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41835670-154e-43c0-b2a8-5609f85e37fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:32.016515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41835670-154e-43c0-b2a8-5609f85e37fc None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.016830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41835670-154e-43c0-b2a8-5609f85e37fc None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.017111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41835670-154e-43c0-b2a8-5609f85e37fc None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.024784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41835670-154e-43c0-b2a8-5609f85e37fc None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:32.026688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41835670-154e-43c0-b2a8-5609f85e37fc None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-41835670-154e-43c0-b2a8-5609f85e37fc None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:32.043221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:32.044213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:32.045110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:32.045110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:32.045110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:32.045110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:32.045110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:32.045110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:32.045110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:32.045110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:32.052059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 468/935] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:32.057116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03736c10-95fa-4ce1-85a7-518ed86a6fe8 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:32.172873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03736c10-95fa-4ce1-85a7-518ed86a6fe8 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:32.206964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36362c3b-f14f-40db-baa7-e574cd84f001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:32.207631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36362c3b-f14f-40db-baa7-e574cd84f001 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.207916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36362c3b-f14f-40db-baa7-e574cd84f001 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.208193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36362c3b-f14f-40db-baa7-e574cd84f001 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.209046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36362c3b-f14f-40db-baa7-e574cd84f001 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:32.209359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36362c3b-f14f-40db-baa7-e574cd84f001 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:32.210803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36362c3b-f14f-40db-baa7-e574cd84f001 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:32.228875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 468/936] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:32.230216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41835670-154e-43c0-b2a8-5609f85e37fc None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:32.324057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:32.343445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41835670-154e-43c0-b2a8-5609f85e37fc None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:32.360686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669d3c15-a7f2-45d6-a70d-574695ec8135 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.361223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669d3c15-a7f2-45d6-a70d-574695ec8135 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.361653 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669d3c15-a7f2-45d6-a70d-574695ec8135 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.363193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 469/937] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:32.368345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855ed90e-5d1a-40df-874d-f3bea8b9f995 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.368741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855ed90e-5d1a-40df-874d-f3bea8b9f995 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.369140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855ed90e-5d1a-40df-874d-f3bea8b9f995 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.386117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d79cf5-63b5-48ad-ad03-96f4004a10df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:32.387291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d79cf5-63b5-48ad-ad03-96f4004a10df None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.387733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d79cf5-63b5-48ad-ad03-96f4004a10df None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.388159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d79cf5-63b5-48ad-ad03-96f4004a10df None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/eb2c6f06b300473ebafff4503bb24396/roles/ee9ee7e94d4e4ac39a965a63e0abea25` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.389517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-855ed90e-5d1a-40df-874d-f3bea8b9f995 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:32.390775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d79cf5-63b5-48ad-ad03-96f4004a10df None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=eb2c6f06b300473ebafff4503bb24396, role_id=ee9ee7e94d4e4ac39a965a63e0abea25)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:32.390981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-855ed90e-5d1a-40df-874d-f3bea8b9f995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:32.392536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d79cf5-63b5-48ad-ad03-96f4004a10df None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:32.423725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:32.461978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 470/938] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:32 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:32.484037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855ed90e-5d1a-40df-874d-f3bea8b9f995 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:32.504184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 469/939] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:46:32 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/eb2c6f06b300473ebafff4503bb24396/roles/ee9ee7e94d4e4ac39a965a63e0abea25 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:46:32.605365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855ed90e-5d1a-40df-874d-f3bea8b9f995 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:32.643780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab3ba07-ecf0-435c-9fd1-96f74f465c66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:32.644612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab3ba07-ecf0-435c-9fd1-96f74f465c66 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.644930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab3ba07-ecf0-435c-9fd1-96f74f465c66 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.645210 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab3ba07-ecf0-435c-9fd1-96f74f465c66 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.645916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab3ba07-ecf0-435c-9fd1-96f74f465c66 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:32.647320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab3ba07-ecf0-435c-9fd1-96f74f465c66 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-aab3ba07-ecf0-435c-9fd1-96f74f465c66 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:32.660686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:32.661816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:32.662835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 471/940] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:32 2022] GET /identity/v3/services/object-store => generated 93 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:32.668115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d79cf5-63b5-48ad-ad03-96f4004a10df None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:32.777259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d79cf5-63b5-48ad-ad03-96f4004a10df None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:32.814993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a64e2927-fbce-4962-baf8-48fa4547536e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:32.815716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64e2927-fbce-4962-baf8-48fa4547536e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.816013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64e2927-fbce-4962-baf8-48fa4547536e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.816295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a64e2927-fbce-4962-baf8-48fa4547536e None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.817030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64e2927-fbce-4962-baf8-48fa4547536e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:32.817339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64e2927-fbce-4962-baf8-48fa4547536e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:32.818840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a64e2927-fbce-4962-baf8-48fa4547536e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:32.825468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 470/941] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:32 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:32.830615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab3ba07-ecf0-435c-9fd1-96f74f465c66 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:32.842385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b7ea02-ea86-44c2-9cdf-3187bf2a881b None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.842675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b7ea02-ea86-44c2-9cdf-3187bf2a881b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.842948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b7ea02-ea86-44c2-9cdf-3187bf2a881b None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.844544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 471/942] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:32.849790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79106484-b274-4571-bc50-b6827818c518 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.850075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79106484-b274-4571-bc50-b6827818c518 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.850357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79106484-b274-4571-bc50-b6827818c518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.867223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-79106484-b274-4571-bc50-b6827818c518 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:32.868141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:32.868408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-79106484-b274-4571-bc50-b6827818c518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:32.870582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:32.945035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 472/943] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:32 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:32.952340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab3ba07-ecf0-435c-9fd1-96f74f465c66 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:32.963357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79106484-b274-4571-bc50-b6827818c518 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:32.993747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319d64b9-d7ac-4589-bd6a-5c1492b8d034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:32.994413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319d64b9-d7ac-4589-bd6a-5c1492b8d034 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:32.996414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319d64b9-d7ac-4589-bd6a-5c1492b8d034 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:32.996414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319d64b9-d7ac-4589-bd6a-5c1492b8d034 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:32.996414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319d64b9-d7ac-4589-bd6a-5c1492b8d034 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:32.997029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319d64b9-d7ac-4589-bd6a-5c1492b8d034 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:33.000792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-319d64b9-d7ac-4589-bd6a-5c1492b8d034 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:33.010043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 472/944] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:32 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:33.015836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319d64b9-d7ac-4589-bd6a-5c1492b8d034 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:33.074291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79106484-b274-4571-bc50-b6827818c518 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:33.110322 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:33.111047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:33.111339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:33.111618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:33.112316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:33.113788 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:33.115194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING keystone.server.flask.common [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jun 10 12:46:33.116156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319d64b9-d7ac-4589-bd6a-5c1492b8d034 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:33.123740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:33.133473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:33.153171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15760115-12a4-4b62-aaa2-ef2605607be9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:33.154126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15760115-12a4-4b62-aaa2-ef2605607be9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:33.154571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15760115-12a4-4b62-aaa2-ef2605607be9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:33.155000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15760115-12a4-4b62-aaa2-ef2605607be9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:33.155944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15760115-12a4-4b62-aaa2-ef2605607be9 None admin] RBAC: Adding query filter params (interface=public, service_id=a6b72841b40647cfb8efe74f3e3ec78a, region_id=RegionOne) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:33.156405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15760115-12a4-4b62-aaa2-ef2605607be9 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:33.157996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15760115-12a4-4b62-aaa2-ef2605607be9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:33.168831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 473/945] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Jun 10 12:46:33 2022] GET /identity/v3/endpoints?service_id=a6b72841b40647cfb8efe74f3e3ec78a&interface=public®ion_id=RegionOne => generated 194 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:33.170519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15760115-12a4-4b62-aaa2-ef2605607be9 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:33.176165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.sql.core [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jun 10 12:46:33.176165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jun 10 12:46:33.176165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [parameters: {'id': '9a3d128903164fa783b5816df85912a1', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jun 10 12:46:33.176165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80299) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 9a3d128903164fa783b5816df85912a1.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 9a3d128903164fa783b5816df85912a1. Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self.dialect.do_execute( Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application result = self._query(query) Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application conn.query(q) Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jun 10 12:46:33.199581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application result.read() Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application packet.raise_for_error() Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jun 10 12:46:33.200456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return project_ref.to_dict() Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application next(self.gen) Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application yield resource Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application next(self.gen) Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self.session.rollback() Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self.force_reraise() Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise self.value Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application session.commit() Jun 10 12:46:33.201485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._prepare_impl() Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self.session.flush() Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._flush(objects) Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application compat.raise_( Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application flush_context.execute() Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jun 10 12:46:33.202382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rec.execute(self) Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application _emit_insert_statements( Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application c = connection._execute_20( Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ret = self._execute_context( Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jun 10 12:46:33.203286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self.dialect.do_execute( Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application result = self._query(query) Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application conn.query(q) Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application result.read() Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jun 10 12:46:33.205836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application packet.raise_for_error() Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [parameters: {'id': '9a3d128903164fa783b5816df85912a1', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:33.207143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:33.208081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.Conflict( Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 9a3d128903164fa783b5816df85912a1. Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 473/946] 10.222.0.25 () {60 vars in 1140 bytes} [Fri Jun 10 12:46:32 2022] POST /identity/v3/projects => generated 262 bytes in 239 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:33.209552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:33.285150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15760115-12a4-4b62-aaa2-ef2605607be9 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:33.319152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ffef91-9db5-497f-818e-b8629bb5aa4e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:33.321224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2292b5-f862-4645-955c-2a6eb0c62fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:33.324207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2292b5-f862-4645-955c-2a6eb0c62fe5 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:33.324801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2292b5-f862-4645-955c-2a6eb0c62fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:33.325286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2292b5-f862-4645-955c-2a6eb0c62fe5 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:33.326172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2292b5-f862-4645-955c-2a6eb0c62fe5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:33.327774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2292b5-f862-4645-955c-2a6eb0c62fe5 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:33.341903 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 474/947] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Jun 10 12:46:33 2022] GET /identity/v3/services => generated 1497 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 12:46:33.357140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7bf494-d7dd-4a85-a493-fab2d0a5954a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:33.357967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7bf494-d7dd-4a85-a493-fab2d0a5954a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:33.358251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7bf494-d7dd-4a85-a493-fab2d0a5954a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:33.358525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7bf494-d7dd-4a85-a493-fab2d0a5954a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:33.367540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7bf494-d7dd-4a85-a493-fab2d0a5954a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:33.369290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7bf494-d7dd-4a85-a493-fab2d0a5954a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-ca7bf494-d7dd-4a85-a493-fab2d0a5954a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:33.385318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:33.386268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:33.387166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:33.387166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:33.387166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:33.387166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:33.387166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:33.387166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:33.387166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:33.387166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:33.387166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 474/948] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:33.392692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2292b5-f862-4645-955c-2a6eb0c62fe5 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:33.422247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:33.503350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2292b5-f862-4645-955c-2a6eb0c62fe5 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:33.539528 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8136bc7e-7f65-41d7-8c1c-23855b54c60d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:33.540515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8136bc7e-7f65-41d7-8c1c-23855b54c60d None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:33.541035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8136bc7e-7f65-41d7-8c1c-23855b54c60d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:33.541530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8136bc7e-7f65-41d7-8c1c-23855b54c60d None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:33.542467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8136bc7e-7f65-41d7-8c1c-23855b54c60d None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:33.542911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8136bc7e-7f65-41d7-8c1c-23855b54c60d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:33.544484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8136bc7e-7f65-41d7-8c1c-23855b54c60d None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:33.562322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 475/949] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:33 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:34.243869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64501949-0829-473a-a768-f0c451911400 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:34.244397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64501949-0829-473a-a768-f0c451911400 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:34.244511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64501949-0829-473a-a768-f0c451911400 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:34.248951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 475/950] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:34 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:34.254187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d999550-cdaa-4173-af17-c8db649d0699 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:34.254615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d999550-cdaa-4173-af17-c8db649d0699 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:34.254750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d999550-cdaa-4173-af17-c8db649d0699 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:34.271907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5d999550-cdaa-4173-af17-c8db649d0699 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:34.273140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5d999550-cdaa-4173-af17-c8db649d0699 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:34.356874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 476/951] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:34 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:34.374021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64501949-0829-473a-a768-f0c451911400 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:34.441543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:34.492591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64501949-0829-473a-a768-f0c451911400 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:34.527421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d0804e-5765-47b2-be02-1d5e1930819b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:34.528136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d0804e-5765-47b2-be02-1d5e1930819b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:34.528436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d0804e-5765-47b2-be02-1d5e1930819b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:34.528691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d0804e-5765-47b2-be02-1d5e1930819b None admin] PATH_INFO: `/v3/services/compute` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:34.529374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d0804e-5765-47b2-be02-1d5e1930819b None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:34.530747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d0804e-5765-47b2-be02-1d5e1930819b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-84d0804e-5765-47b2-be02-1d5e1930819b None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:34.543123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:34.544218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:34.545159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 476/952] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:34 2022] GET /identity/v3/services/compute => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:34.550251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d999550-cdaa-4173-af17-c8db649d0699 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:34.565419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:34.652167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d999550-cdaa-4173-af17-c8db649d0699 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:34.686083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdec32ac-b979-4c3a-9476-620be22eedb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:34.686743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdec32ac-b979-4c3a-9476-620be22eedb9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:34.687029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdec32ac-b979-4c3a-9476-620be22eedb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:34.687307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdec32ac-b979-4c3a-9476-620be22eedb9 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:34.688010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdec32ac-b979-4c3a-9476-620be22eedb9 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:34.688318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdec32ac-b979-4c3a-9476-620be22eedb9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:34.689723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdec32ac-b979-4c3a-9476-620be22eedb9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:34.695535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 477/953] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:34 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:34.701863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d0804e-5765-47b2-be02-1d5e1930819b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:34.789438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d1586c-fdeb-4980-a6d2-8b44af04a5bf None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:34.789790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d1586c-fdeb-4980-a6d2-8b44af04a5bf None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:34.790173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d1586c-fdeb-4980-a6d2-8b44af04a5bf None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:34.791867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 478/954] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:34.797557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6724f55-5fe7-4ef2-a899-32efd1da8519 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:34.797557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6724f55-5fe7-4ef2-a899-32efd1da8519 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:34.797557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6724f55-5fe7-4ef2-a899-32efd1da8519 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:34.818801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d0804e-5765-47b2-be02-1d5e1930819b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:34.822305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e6724f55-5fe7-4ef2-a899-32efd1da8519 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:34.823528 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e6724f55-5fe7-4ef2-a899-32efd1da8519 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:34.839924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:34.860193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c04f422-fdcb-44e3-a9b5-210e8d40c904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:34.861287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04f422-fdcb-44e3-a9b5-210e8d40c904 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:34.861738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04f422-fdcb-44e3-a9b5-210e8d40c904 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:34.862446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c04f422-fdcb-44e3-a9b5-210e8d40c904 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:34.863566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c04f422-fdcb-44e3-a9b5-210e8d40c904 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:34.864027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c04f422-fdcb-44e3-a9b5-210e8d40c904 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:34.866212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c04f422-fdcb-44e3-a9b5-210e8d40c904 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:34.880792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 477/955] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:34 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:34.913686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 479/956] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:34 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:34.937607 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c04f422-fdcb-44e3-a9b5-210e8d40c904 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:35.053942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c04f422-fdcb-44e3-a9b5-210e8d40c904 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:35.089029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561b9694-2198-4e42-8604-2917275b68c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:35.089846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561b9694-2198-4e42-8604-2917275b68c0 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.090131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561b9694-2198-4e42-8604-2917275b68c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.090415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561b9694-2198-4e42-8604-2917275b68c0 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.101528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561b9694-2198-4e42-8604-2917275b68c0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-561b9694-2198-4e42-8604-2917275b68c0 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:35.109842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:35.110808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:35.111719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 478/957] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Jun 10 12:46:34 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:35.119367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6724f55-5fe7-4ef2-a899-32efd1da8519 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:35.130092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:35.230013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6724f55-5fe7-4ef2-a899-32efd1da8519 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:35.265963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7aa082-4cac-4e7a-9e1e-6dc5199ad8ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:35.266727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7aa082-4cac-4e7a-9e1e-6dc5199ad8ac None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.266911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7aa082-4cac-4e7a-9e1e-6dc5199ad8ac None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.267196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7aa082-4cac-4e7a-9e1e-6dc5199ad8ac None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.267954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7aa082-4cac-4e7a-9e1e-6dc5199ad8ac None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:35.268266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7aa082-4cac-4e7a-9e1e-6dc5199ad8ac None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:35.269686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7aa082-4cac-4e7a-9e1e-6dc5199ad8ac None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:35.279151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 480/958] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Jun 10 12:46:35 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:35.284564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561b9694-2198-4e42-8604-2917275b68c0 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:35.337486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:35.397344 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561b9694-2198-4e42-8604-2917275b68c0 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:35.432483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5665bd44-0a42-467a-a8eb-b7414582e7d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:35.433314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5665bd44-0a42-467a-a8eb-b7414582e7d5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.433599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5665bd44-0a42-467a-a8eb-b7414582e7d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.433879 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5665bd44-0a42-467a-a8eb-b7414582e7d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.445441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5665bd44-0a42-467a-a8eb-b7414582e7d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:35.446911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5665bd44-0a42-467a-a8eb-b7414582e7d5 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-5665bd44-0a42-467a-a8eb-b7414582e7d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:35.466841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:35.467782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:35.468667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:35.468667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:35.468667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:35.468667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:35.468667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 479/959] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:35.475163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7aa082-4cac-4e7a-9e1e-6dc5199ad8ac None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:35.551822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d5065e-706d-4db1-94ff-e0d92669eb4b None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.552692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d5065e-706d-4db1-94ff-e0d92669eb4b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.552959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d5065e-706d-4db1-94ff-e0d92669eb4b None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.554619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 480/960] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:35.559772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:35.560374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ce801e-c7f2-45d4-ac12-9565cb1880d3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.561538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ce801e-c7f2-45d4-ac12-9565cb1880d3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.562036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ce801e-c7f2-45d4-ac12-9565cb1880d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.579694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-29ce801e-c7f2-45d4-ac12-9565cb1880d3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:35.581744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-29ce801e-c7f2-45d4-ac12-9565cb1880d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:35.593039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7aa082-4cac-4e7a-9e1e-6dc5199ad8ac None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:35.628209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d0eeac-32f9-4065-bf90-cbd423bdf512 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:35.628935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d0eeac-32f9-4065-bf90-cbd423bdf512 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.629225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d0eeac-32f9-4065-bf90-cbd423bdf512 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.629504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d0eeac-32f9-4065-bf90-cbd423bdf512 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.630262 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d0eeac-32f9-4065-bf90-cbd423bdf512 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:35.630571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d0eeac-32f9-4065-bf90-cbd423bdf512 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:35.631941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d0eeac-32f9-4065-bf90-cbd423bdf512 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:35.644423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 481/961] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:35.650416 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d0eeac-32f9-4065-bf90-cbd423bdf512 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:35.659484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 481/962] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:35 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:35.689442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ce801e-c7f2-45d4-ac12-9565cb1880d3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:35.718031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:35.769886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d0eeac-32f9-4065-bf90-cbd423bdf512 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:35.800868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:35.804825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ce801e-c7f2-45d4-ac12-9565cb1880d3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:35.807133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52614c24-d185-4f9d-960e-146d3891197b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:35.808076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52614c24-d185-4f9d-960e-146d3891197b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.808223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52614c24-d185-4f9d-960e-146d3891197b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.808503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52614c24-d185-4f9d-960e-146d3891197b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.815794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52614c24-d185-4f9d-960e-146d3891197b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:35.817282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52614c24-d185-4f9d-960e-146d3891197b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-52614c24-d185-4f9d-960e-146d3891197b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:35.833373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:35.834298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:35.835224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 482/963] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:46:35 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:35.835831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0551325-e6d9-4f7b-97e3-2d4153e4da05 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.836243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0551325-e6d9-4f7b-97e3-2d4153e4da05 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.836333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0551325-e6d9-4f7b-97e3-2d4153e4da05 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.837750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 483/964] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:35.839876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0551325-e6d9-4f7b-97e3-2d4153e4da05 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:35.852565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a640ffba-030d-47b3-9c20-b2f5bdf5efa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:35.853634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a640ffba-030d-47b3-9c20-b2f5bdf5efa3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.854145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a640ffba-030d-47b3-9c20-b2f5bdf5efa3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.854696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a640ffba-030d-47b3-9c20-b2f5bdf5efa3 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.855716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a640ffba-030d-47b3-9c20-b2f5bdf5efa3 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:35.857319 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a640ffba-030d-47b3-9c20-b2f5bdf5efa3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-a640ffba-030d-47b3-9c20-b2f5bdf5efa3 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:35.871005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:35.871979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:35.872844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:35.879083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 482/965] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:35 2022] GET /identity/v3/services/object-store => generated 93 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:35.879083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5e4466-133b-4f6c-b989-626cfd12bb1e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:35.879083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5e4466-133b-4f6c-b989-626cfd12bb1e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:35.879083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5e4466-133b-4f6c-b989-626cfd12bb1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:35.893047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-7e5e4466-133b-4f6c-b989-626cfd12bb1e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:35.894241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7e5e4466-133b-4f6c-b989-626cfd12bb1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:35.946994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0551325-e6d9-4f7b-97e3-2d4153e4da05 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:35.962261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 483/966] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:35 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:35.962972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5e4466-133b-4f6c-b989-626cfd12bb1e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:35.981330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:36.001732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add3ebe6-224c-4331-a23a-f1193e61a067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.002707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3ebe6-224c-4331-a23a-f1193e61a067 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.002990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3ebe6-224c-4331-a23a-f1193e61a067 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.003266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add3ebe6-224c-4331-a23a-f1193e61a067 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.006190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add3ebe6-224c-4331-a23a-f1193e61a067 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:36.006511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add3ebe6-224c-4331-a23a-f1193e61a067 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.008807 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add3ebe6-224c-4331-a23a-f1193e61a067 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.031984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 484/967] 10.222.0.25 () {56 vars in 1149 bytes} [Fri Jun 10 12:46:35 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:36.032824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add3ebe6-224c-4331-a23a-f1193e61a067 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:36.074581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5e4466-133b-4f6c-b989-626cfd12bb1e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:36.089872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:36.111141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47546e2c-0f6a-4c92-8b67-32d9d8e68639 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.111729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47546e2c-0f6a-4c92-8b67-32d9d8e68639 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.112021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47546e2c-0f6a-4c92-8b67-32d9d8e68639 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.112279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47546e2c-0f6a-4c92-8b67-32d9d8e68639 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.113018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47546e2c-0f6a-4c92-8b67-32d9d8e68639 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:36.113375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47546e2c-0f6a-4c92-8b67-32d9d8e68639 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.114778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47546e2c-0f6a-4c92-8b67-32d9d8e68639 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.120062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 484/968] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:35 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:36.120866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47546e2c-0f6a-4c92-8b67-32d9d8e68639 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:36.147417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add3ebe6-224c-4331-a23a-f1193e61a067 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:36.181573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.182258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.182582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.182870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.183568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.185078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.186380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WARNING keystone.server.flask.common [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jun 10 12:46:36.212147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:36.229417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47546e2c-0f6a-4c92-8b67-32d9d8e68639 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:36.238837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.sql.core [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jun 10 12:46:36.238837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jun 10 12:46:36.238837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [parameters: {'user_id': '204f09b4a8cb46b0b139ef8581c92ca4', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jun 10 12:46:36.238837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80300) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.dialect.do_execute( Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application result = self._query(query) Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application conn.query(q) Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application result.read() Jun 10 12:46:36.247017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application packet.raise_for_error() Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jun 10 12:46:36.247902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application next(self.gen) Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application yield resource Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application next(self.gen) Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.session.rollback() Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.force_reraise() Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise self.value Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application session.commit() Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jun 10 12:46:36.248931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._prepare_impl() Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.session.flush() Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._flush(objects) Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application compat.raise_( Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application flush_context.execute() Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rec.execute(self) Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jun 10 12:46:36.249835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application _emit_insert_statements( Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application result = connection._execute_20( Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ret = self._execute_context( Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jun 10 12:46:36.250728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.dialect.do_execute( Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application result = self._query(query) Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application conn.query(q) Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application result.read() Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jun 10 12:46:36.251574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application packet.raise_for_error() Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [parameters: {'user_id': '204f09b4a8cb46b0b139ef8581c92ca4', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:36.252434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jun 10 12:46:36.254455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jun 10 12:46:36.256266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jun 10 12:46:36.256266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jun 10 12:46:36.256266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jun 10 12:46:36.256266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jun 10 12:46:36.256266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jun 10 12:46:36.256266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:36.256266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 485/969] 10.222.0.25 () {60 vars in 1131 bytes} [Fri Jun 10 12:46:36 2022] POST /identity/v3/users => generated 160 bytes in 216 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:36.256266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:36.276366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2475c059-f414-4613-9738-799c512be675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.277066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475c059-f414-4613-9738-799c512be675 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.277349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475c059-f414-4613-9738-799c512be675 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.277630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475c059-f414-4613-9738-799c512be675 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.358571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2475c059-f414-4613-9738-799c512be675 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.359997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43766a20-4c12-4df7-ad06-be03c0b00fff None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:36.360368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2475c059-f414-4613-9738-799c512be675 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.373665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:36.399198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64cf2e1-0497-40fd-bc52-c2f8ad018efa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.404652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64cf2e1-0497-40fd-bc52-c2f8ad018efa None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.404652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64cf2e1-0497-40fd-bc52-c2f8ad018efa None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.404652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64cf2e1-0497-40fd-bc52-c2f8ad018efa None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.404652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64cf2e1-0497-40fd-bc52-c2f8ad018efa None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:36.404652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64cf2e1-0497-40fd-bc52-c2f8ad018efa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.404652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64cf2e1-0497-40fd-bc52-c2f8ad018efa None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.411007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 486/970] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:36 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:36.411897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64cf2e1-0497-40fd-bc52-c2f8ad018efa None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:36.458817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:36.478031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 485/971] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:36 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:46:36.478832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2475c059-f414-4613-9738-799c512be675 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:36.519185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64cf2e1-0497-40fd-bc52-c2f8ad018efa None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:36.554910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711bf10a-620e-4f9c-a13e-365b29eee864 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.555798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711bf10a-620e-4f9c-a13e-365b29eee864 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.556086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711bf10a-620e-4f9c-a13e-365b29eee864 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.556365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-711bf10a-620e-4f9c-a13e-365b29eee864 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.564043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711bf10a-620e-4f9c-a13e-365b29eee864 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.565509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-711bf10a-620e-4f9c-a13e-365b29eee864 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-711bf10a-620e-4f9c-a13e-365b29eee864 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:36.579435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:36.580417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 487/972] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:36.581411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711bf10a-620e-4f9c-a13e-365b29eee864 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:36.591221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2475c059-f414-4613-9738-799c512be675 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:36.620973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:36.628886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e9e539-e0fe-49a4-949c-6d6e9e748553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.629485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e9e539-e0fe-49a4-949c-6d6e9e748553 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.629764 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e9e539-e0fe-49a4-949c-6d6e9e748553 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.630579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e9e539-e0fe-49a4-949c-6d6e9e748553 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.631323 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e9e539-e0fe-49a4-949c-6d6e9e748553 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.632696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e9e539-e0fe-49a4-949c-6d6e9e748553 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.689064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-711bf10a-620e-4f9c-a13e-365b29eee864 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:36.713466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 486/973] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Jun 10 12:46:36 2022] POST /identity/v3/endpoints => generated 349 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:46:36.721968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e9e539-e0fe-49a4-949c-6d6e9e748553 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:36.731832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b280b9-4ba9-4410-b8c8-abe48fe70ca0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.732648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b280b9-4ba9-4410-b8c8-abe48fe70ca0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.732959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b280b9-4ba9-4410-b8c8-abe48fe70ca0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.733242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b280b9-4ba9-4410-b8c8-abe48fe70ca0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.746029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b280b9-4ba9-4410-b8c8-abe48fe70ca0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.747554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b280b9-4ba9-4410-b8c8-abe48fe70ca0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-22b280b9-4ba9-4410-b8c8-abe48fe70ca0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:36.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:36.768725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:36.769650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:36.769650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:36.769650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:36.769650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:36.769650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 488/974] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:36.777055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b280b9-4ba9-4410-b8c8-abe48fe70ca0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:36.783191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:36.877496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:36.879568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e9e539-e0fe-49a4-949c-6d6e9e748553 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:36.899055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b280b9-4ba9-4410-b8c8-abe48fe70ca0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:36.923740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923efd33-a110-44bb-9ce7-9608221d1e35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.924418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923efd33-a110-44bb-9ce7-9608221d1e35 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.924700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923efd33-a110-44bb-9ce7-9608221d1e35 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.925024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923efd33-a110-44bb-9ce7-9608221d1e35 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.925830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923efd33-a110-44bb-9ce7-9608221d1e35 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:36.926141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923efd33-a110-44bb-9ce7-9608221d1e35 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.927645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923efd33-a110-44bb-9ce7-9608221d1e35 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.939214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e58398-1c40-4e33-86d3-eab3cfba91ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:36.939575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e58398-1c40-4e33-86d3-eab3cfba91ce None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:36.939859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e58398-1c40-4e33-86d3-eab3cfba91ce None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:36.940135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e58398-1c40-4e33-86d3-eab3cfba91ce None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:36.940897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e58398-1c40-4e33-86d3-eab3cfba91ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:36.941204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e58398-1c40-4e33-86d3-eab3cfba91ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:36.943181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e58398-1c40-4e33-86d3-eab3cfba91ce None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:36.956404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 489/975] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:36.962330 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e58398-1c40-4e33-86d3-eab3cfba91ce None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:36.969309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 487/976] 10.222.0.25 () {56 vars in 1101 bytes} [Fri Jun 10 12:46:36 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:37.073278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e58398-1c40-4e33-86d3-eab3cfba91ce None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:37.112060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e4b972-c5ce-4472-9a4c-c844955057f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:37.112898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e4b972-c5ce-4472-9a4c-c844955057f2 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:37.113182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e4b972-c5ce-4472-9a4c-c844955057f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:37.113458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e4b972-c5ce-4472-9a4c-c844955057f2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:37.121578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e4b972-c5ce-4472-9a4c-c844955057f2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:37.123128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e4b972-c5ce-4472-9a4c-c844955057f2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c2e4b972-c5ce-4472-9a4c-c844955057f2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:37.139754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:37.140961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:37.142067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:37.142067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:37.142067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:37.142067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:37.142067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:37.142067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:37.142067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:37.142067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:37.142067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 490/977] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:37.144048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac355ed1-6199-4a21-9a03-19f01dc014ef None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:37.144349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac355ed1-6199-4a21-9a03-19f01dc014ef None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:37.144633 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac355ed1-6199-4a21-9a03-19f01dc014ef None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:37.146179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 488/978] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:37.148454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e4b972-c5ce-4472-9a4c-c844955057f2 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:37.151910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:37.152079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce966856-c44e-4b64-8eb0-8be1d7682822 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:37.152207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce966856-c44e-4b64-8eb0-8be1d7682822 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:37.152497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce966856-c44e-4b64-8eb0-8be1d7682822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:37.170274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ce966856-c44e-4b64-8eb0-8be1d7682822 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:37.171509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ce966856-c44e-4b64-8eb0-8be1d7682822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:37.265131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 489/979] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:37 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:37.277215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e4b972-c5ce-4472-9a4c-c844955057f2 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:37.288376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce966856-c44e-4b64-8eb0-8be1d7682822 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:37.318041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a26bd9-b7e9-42e0-a015-715e1df91c60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:37.318720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a26bd9-b7e9-42e0-a015-715e1df91c60 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:37.319009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a26bd9-b7e9-42e0-a015-715e1df91c60 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:37.319287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a26bd9-b7e9-42e0-a015-715e1df91c60 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:37.320077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a26bd9-b7e9-42e0-a015-715e1df91c60 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:37.320387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a26bd9-b7e9-42e0-a015-715e1df91c60 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:37.321873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a26bd9-b7e9-42e0-a015-715e1df91c60 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:37.325851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:37.339988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 491/980] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:37.346083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a26bd9-b7e9-42e0-a015-715e1df91c60 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:37.406992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce966856-c44e-4b64-8eb0-8be1d7682822 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:37.418606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:37.443252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9adbc5cd-d723-4df3-af30-32d930b43834 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:37.444179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adbc5cd-d723-4df3-af30-32d930b43834 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:37.444644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adbc5cd-d723-4df3-af30-32d930b43834 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:37.445108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9adbc5cd-d723-4df3-af30-32d930b43834 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:37.446018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adbc5cd-d723-4df3-af30-32d930b43834 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:37.447545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9adbc5cd-d723-4df3-af30-32d930b43834 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:37.460464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a26bd9-b7e9-42e0-a015-715e1df91c60 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:37.495135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5dbd1c1-66fa-477c-9875-7b0d5c7a8fdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:37.495780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dbd1c1-66fa-477c-9875-7b0d5c7a8fdf None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:37.496063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dbd1c1-66fa-477c-9875-7b0d5c7a8fdf None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:37.496341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5dbd1c1-66fa-477c-9875-7b0d5c7a8fdf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:37.497197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dbd1c1-66fa-477c-9875-7b0d5c7a8fdf None admin] RBAC: Adding query filter params (role.id=ee9ee7e94d4e4ac39a965a63e0abea25, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=eb2c6f06b300473ebafff4503bb24396) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:37.497524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dbd1c1-66fa-477c-9875-7b0d5c7a8fdf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:37.498936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5dbd1c1-66fa-477c-9875-7b0d5c7a8fdf None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:37.504576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 492/981] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:37 2022] GET /identity/v3/role_assignments?user.id=eb2c6f06b300473ebafff4503bb24396&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=ee9ee7e94d4e4ac39a965a63e0abea25 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:37.543494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 490/982] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:46:37 2022] POST /identity/v3/services => generated 242 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:46:39.031877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16034410-219c-4cc0-b8b2-d989b299e0c9 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.032428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16034410-219c-4cc0-b8b2-d989b299e0c9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.032428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16034410-219c-4cc0-b8b2-d989b299e0c9 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.033926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 493/983] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:39.042162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e7c469-73e8-4e9e-937f-e71e5e774894 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.042585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e7c469-73e8-4e9e-937f-e71e5e774894 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.043442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e7c469-73e8-4e9e-937f-e71e5e774894 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.061056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-82e7c469-73e8-4e9e-937f-e71e5e774894 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:39.062643 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-82e7c469-73e8-4e9e-937f-e71e5e774894 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:39.146710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 491/984] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:39 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:39.169780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16034410-219c-4cc0-b8b2-d989b299e0c9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:39.269429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16034410-219c-4cc0-b8b2-d989b299e0c9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:39.295098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:39.303840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a13150-1d9c-47d9-9aff-fbce82e053e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:39.307916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a13150-1d9c-47d9-9aff-fbce82e053e3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.307916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a13150-1d9c-47d9-9aff-fbce82e053e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.307916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a13150-1d9c-47d9-9aff-fbce82e053e3 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.307916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a13150-1d9c-47d9-9aff-fbce82e053e3 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:39.307916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a13150-1d9c-47d9-9aff-fbce82e053e3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c9a13150-1d9c-47d9-9aff-fbce82e053e3 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:39.321983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:39.324815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:39.325701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 494/985] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:39 2022] GET /identity/v3/services/object-store => generated 93 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:39.327646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e7c469-73e8-4e9e-937f-e71e5e774894 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:39.346481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:39.436408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e7c469-73e8-4e9e-937f-e71e5e774894 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:39.469184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799a061a-e447-4511-8409-030aed4711d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:39.469844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799a061a-e447-4511-8409-030aed4711d8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.470124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799a061a-e447-4511-8409-030aed4711d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.470399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799a061a-e447-4511-8409-030aed4711d8 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.471097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799a061a-e447-4511-8409-030aed4711d8 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:39.471403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799a061a-e447-4511-8409-030aed4711d8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:39.472790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799a061a-e447-4511-8409-030aed4711d8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:39.482109 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 492/986] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:39 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:39.483704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a13150-1d9c-47d9-9aff-fbce82e053e3 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:39.587174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a13150-1d9c-47d9-9aff-fbce82e053e3 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:39.623872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2da4ef7-f308-4ab0-a64c-a49887691cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:39.624573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da4ef7-f308-4ab0-a64c-a49887691cb7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.624894 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da4ef7-f308-4ab0-a64c-a49887691cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.625177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2da4ef7-f308-4ab0-a64c-a49887691cb7 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.625929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2da4ef7-f308-4ab0-a64c-a49887691cb7 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:39.626243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2da4ef7-f308-4ab0-a64c-a49887691cb7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:39.627716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2da4ef7-f308-4ab0-a64c-a49887691cb7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:39.634011 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 495/987] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:39 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:39.640266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799a061a-e447-4511-8409-030aed4711d8 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:39.687801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780b5397-d2ce-43a2-a17d-bacfeaf49c79 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.688088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780b5397-d2ce-43a2-a17d-bacfeaf49c79 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.688369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780b5397-d2ce-43a2-a17d-bacfeaf49c79 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.689910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 496/988] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:39.690791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449467b2-d9d3-4e38-bba4-35932cbe0baa None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.691071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449467b2-d9d3-4e38-bba4-35932cbe0baa None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.691343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449467b2-d9d3-4e38-bba4-35932cbe0baa None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.692668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 497/989] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:39 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:39.694594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f97a9-9e8e-47aa-a0a5-7d4fb3628595 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.694886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f97a9-9e8e-47aa-a0a5-7d4fb3628595 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.695165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-158f97a9-9e8e-47aa-a0a5-7d4fb3628595 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.711614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-158f97a9-9e8e-47aa-a0a5-7d4fb3628595 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:39.713141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-158f97a9-9e8e-47aa-a0a5-7d4fb3628595 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:39.720504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:39.743671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799a061a-e447-4511-8409-030aed4711d8 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:39.761425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:39.778928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 498/990] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:39 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:39.780252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab8ae43-ca54-482d-8d1a-37a9d2573026 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.780539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab8ae43-ca54-482d-8d1a-37a9d2573026 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.780845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab8ae43-ca54-482d-8d1a-37a9d2573026 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.787363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb0ac26-5efd-41f1-bc06-d4428198bc8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:39.788104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb0ac26-5efd-41f1-bc06-d4428198bc8f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.788442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb0ac26-5efd-41f1-bc06-d4428198bc8f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.788726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb0ac26-5efd-41f1-bc06-d4428198bc8f None admin] PATH_INFO: `/v3/endpoints` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.789565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb0ac26-5efd-41f1-bc06-d4428198bc8f None admin] RBAC: Adding query filter params (interface=admin, service_id=a6b72841b40647cfb8efe74f3e3ec78a, region_id=RegionOne) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:39.789881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb0ac26-5efd-41f1-bc06-d4428198bc8f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:39.791267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcb0ac26-5efd-41f1-bc06-d4428198bc8f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:39.801441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 493/991] 10.222.0.25 () {56 vars in 1249 bytes} [Fri Jun 10 12:46:39 2022] GET /identity/v3/endpoints?service_id=a6b72841b40647cfb8efe74f3e3ec78a&interface=admin®ion_id=RegionOne => generated 193 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:39.802548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb0ac26-5efd-41f1-bc06-d4428198bc8f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:39.805162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-eab8ae43-ca54-482d-8d1a-37a9d2573026 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:39.806342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-eab8ae43-ca54-482d-8d1a-37a9d2573026 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:39.872667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 499/992] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:39 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:39.873527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab8ae43-ca54-482d-8d1a-37a9d2573026 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:39.903714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb0ac26-5efd-41f1-bc06-d4428198bc8f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:39.937217 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81ccd41-8c4b-447c-81d3-507266824258 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:39.938003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81ccd41-8c4b-447c-81d3-507266824258 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:39.938285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81ccd41-8c4b-447c-81d3-507266824258 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:39.938558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81ccd41-8c4b-447c-81d3-507266824258 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:39.939202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81ccd41-8c4b-447c-81d3-507266824258 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:39.940572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81ccd41-8c4b-447c-81d3-507266824258 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-d81ccd41-8c4b-447c-81d3-507266824258 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:39.952495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:39.958645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:39.959564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 494/993] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:39 2022] GET /identity/v3/services/compute => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:39.960341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81ccd41-8c4b-447c-81d3-507266824258 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:39.960341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:39.973317 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab8ae43-ca54-482d-8d1a-37a9d2573026 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:39.987043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:40.006868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f86f5e5-83a4-43a3-af2e-02b91bad071d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:40.007523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f86f5e5-83a4-43a3-af2e-02b91bad071d None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:40.007809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f86f5e5-83a4-43a3-af2e-02b91bad071d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:40.008084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f86f5e5-83a4-43a3-af2e-02b91bad071d None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:40.008755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f86f5e5-83a4-43a3-af2e-02b91bad071d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:40.010187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f86f5e5-83a4-43a3-af2e-02b91bad071d None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:40.016812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 500/994] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Jun 10 12:46:39 2022] GET /identity/v3/services => generated 1727 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 12:46:40.017879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f86f5e5-83a4-43a3-af2e-02b91bad071d None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:40.061561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81ccd41-8c4b-447c-81d3-507266824258 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:40.100776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd3572f-5b1b-4f77-aa2f-7f7c2f6eef07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:40.101603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd3572f-5b1b-4f77-aa2f-7f7c2f6eef07 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:40.101892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd3572f-5b1b-4f77-aa2f-7f7c2f6eef07 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:40.102170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd3572f-5b1b-4f77-aa2f-7f7c2f6eef07 None admin] PATH_INFO: `/v3/services/image` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:40.102834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd3572f-5b1b-4f77-aa2f-7f7c2f6eef07 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:40.104248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd3572f-5b1b-4f77-aa2f-7f7c2f6eef07 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-1fd3572f-5b1b-4f77-aa2f-7f7c2f6eef07 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:40.117648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:40.118706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:40.119546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 495/995] 10.222.0.25 () {56 vars in 1105 bytes} [Fri Jun 10 12:46:39 2022] GET /identity/v3/services/image => generated 86 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:40.131333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd3572f-5b1b-4f77-aa2f-7f7c2f6eef07 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:40.133536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f86f5e5-83a4-43a3-af2e-02b91bad071d None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:40.171616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dcec1b-49a7-438b-9d2f-a8b4eb657498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:40.172322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dcec1b-49a7-438b-9d2f-a8b4eb657498 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:40.172569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dcec1b-49a7-438b-9d2f-a8b4eb657498 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:40.172869 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93dcec1b-49a7-438b-9d2f-a8b4eb657498 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:40.173579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dcec1b-49a7-438b-9d2f-a8b4eb657498 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:40.173881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dcec1b-49a7-438b-9d2f-a8b4eb657498 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:40.175335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93dcec1b-49a7-438b-9d2f-a8b4eb657498 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:40.181522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 501/996] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:40 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:40.186830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dcec1b-49a7-438b-9d2f-a8b4eb657498 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:40.204316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:40.242146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:40.246249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd3572f-5b1b-4f77-aa2f-7f7c2f6eef07 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:40.281434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23be107-029f-44a9-96a0-96f5179a55cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:40.282081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23be107-029f-44a9-96a0-96f5179a55cd None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:40.282599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23be107-029f-44a9-96a0-96f5179a55cd None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:40.282599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23be107-029f-44a9-96a0-96f5179a55cd None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:40.283350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23be107-029f-44a9-96a0-96f5179a55cd None admin] RBAC: Adding query filter params (name=image) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:40.283618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23be107-029f-44a9-96a0-96f5179a55cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:40.285052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b23be107-029f-44a9-96a0-96f5179a55cd None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:40.290349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 496/997] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:40 2022] GET /identity/v3/services?name=image => generated 122 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:40.296252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23be107-029f-44a9-96a0-96f5179a55cd None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:40.305922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93dcec1b-49a7-438b-9d2f-a8b4eb657498 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:40.348826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8319ea3b-9740-4340-b187-e81b75429861 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:40.349538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8319ea3b-9740-4340-b187-e81b75429861 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:40.349825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8319ea3b-9740-4340-b187-e81b75429861 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:40.350104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8319ea3b-9740-4340-b187-e81b75429861 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:40.350808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8319ea3b-9740-4340-b187-e81b75429861 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:40.351193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8319ea3b-9740-4340-b187-e81b75429861 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:40.352564 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8319ea3b-9740-4340-b187-e81b75429861 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:40.362730 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 502/998] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:40 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:40.363767 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8319ea3b-9740-4340-b187-e81b75429861 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:40.411485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23be107-029f-44a9-96a0-96f5179a55cd None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:40.445456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:40.451315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e51c87d-2ee6-4ac0-bca8-fb101277f1dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:40.451987 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e51c87d-2ee6-4ac0-bca8-fb101277f1dd None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:40.452274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e51c87d-2ee6-4ac0-bca8-fb101277f1dd None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:40.452552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e51c87d-2ee6-4ac0-bca8-fb101277f1dd None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:40.453407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e51c87d-2ee6-4ac0-bca8-fb101277f1dd None admin] RBAC: Adding query filter params (type=image) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:40.453705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e51c87d-2ee6-4ac0-bca8-fb101277f1dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:40.455205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e51c87d-2ee6-4ac0-bca8-fb101277f1dd None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:40.468806 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 497/999] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:40 2022] GET /identity/v3/services?type=image => generated 122 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:40.479707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8319ea3b-9740-4340-b187-e81b75429861 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:40.488735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:40.519586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a59a73f-d901-4391-b898-4587d1c73f74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:40.520266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a59a73f-d901-4391-b898-4587d1c73f74 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:40.521199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a59a73f-d901-4391-b898-4587d1c73f74 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:40.521556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a59a73f-d901-4391-b898-4587d1c73f74 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:40.522377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a59a73f-d901-4391-b898-4587d1c73f74 None admin] RBAC: Adding query filter params (interface=public, service_id=c6d94259151142569a39d7d24efb8d36, region_id=RegionOne) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:40.522690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a59a73f-d901-4391-b898-4587d1c73f74 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:40.524235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a59a73f-d901-4391-b898-4587d1c73f74 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:40.535703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 503/1000] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Jun 10 12:46:40 2022] GET /identity/v3/endpoints?service_id=c6d94259151142569a39d7d24efb8d36&interface=public®ion_id=RegionOne => generated 194 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:40.537505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e51c87d-2ee6-4ac0-bca8-fb101277f1dd None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:40.646803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e51c87d-2ee6-4ac0-bca8-fb101277f1dd None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:40.681047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec3e522-383f-44e5-b996-308a4541af3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:40.681643 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec3e522-383f-44e5-b996-308a4541af3f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:40.681930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec3e522-383f-44e5-b996-308a4541af3f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:40.682211 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec3e522-383f-44e5-b996-308a4541af3f None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:40.682889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec3e522-383f-44e5-b996-308a4541af3f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:40.684263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec3e522-383f-44e5-b996-308a4541af3f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:40.690715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 498/1001] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Jun 10 12:46:40 2022] GET /identity/v3/services => generated 1727 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 12:46:42.184621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375a5ddf-820c-47bf-b93b-6875d1bbb5cb None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.185180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375a5ddf-820c-47bf-b93b-6875d1bbb5cb None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.185290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-375a5ddf-820c-47bf-b93b-6875d1bbb5cb None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.186624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 504/1002] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:42.191819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e278d0b-84c4-475a-af9a-0bf1e1bb8ed0 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.192720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e278d0b-84c4-475a-af9a-0bf1e1bb8ed0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.192973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e278d0b-84c4-475a-af9a-0bf1e1bb8ed0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.209516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-2e278d0b-84c4-475a-af9a-0bf1e1bb8ed0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:42.210706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2e278d0b-84c4-475a-af9a-0bf1e1bb8ed0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:42.266687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:42.279259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 499/1003] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:42 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:42.302321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375a5ddf-820c-47bf-b93b-6875d1bbb5cb None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:42.414992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-375a5ddf-820c-47bf-b93b-6875d1bbb5cb None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:42.449772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec801cc8-649c-4bd0-9768-bfc537579757 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:42.450576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec801cc8-649c-4bd0-9768-bfc537579757 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.450860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec801cc8-649c-4bd0-9768-bfc537579757 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.451136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec801cc8-649c-4bd0-9768-bfc537579757 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.451791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec801cc8-649c-4bd0-9768-bfc537579757 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:42.453193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec801cc8-649c-4bd0-9768-bfc537579757 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-ec801cc8-649c-4bd0-9768-bfc537579757 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:42.465656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:42.466647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jun 10 12:46:42.467575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:42.471829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 505/1004] 10.222.0.25 () {56 vars in 1126 bytes} [Fri Jun 10 12:46:42 2022] GET /identity/v3/services/object-store => generated 93 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:42.473218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e278d0b-84c4-475a-af9a-0bf1e1bb8ed0 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:42.579281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e278d0b-84c4-475a-af9a-0bf1e1bb8ed0 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:42.615411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9ef128d-d970-49b3-9650-556b9f342d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:42.616078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef128d-d970-49b3-9650-556b9f342d2d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.616363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef128d-d970-49b3-9650-556b9f342d2d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.616642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9ef128d-d970-49b3-9650-556b9f342d2d None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.617384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ef128d-d970-49b3-9650-556b9f342d2d None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:42.617695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ef128d-d970-49b3-9650-556b9f342d2d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:42.619074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9ef128d-d970-49b3-9650-556b9f342d2d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:42.624464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60203294-428f-44af-966a-362350d56bf2 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.624759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60203294-428f-44af-966a-362350d56bf2 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.625082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60203294-428f-44af-966a-362350d56bf2 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.626592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 506/1006] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:42.628838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 500/1006] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:42 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:42.630768 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60203294-428f-44af-966a-362350d56bf2 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:42.631934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08222824-1f0d-432f-9fda-289514bb2fbd None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.632224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08222824-1f0d-432f-9fda-289514bb2fbd None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.632502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08222824-1f0d-432f-9fda-289514bb2fbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.649257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-08222824-1f0d-432f-9fda-289514bb2fbd None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:42.650702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-08222824-1f0d-432f-9fda-289514bb2fbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:42.668902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:42.706765 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:42.724659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 501/1007] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:42 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:42.739225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60203294-428f-44af-966a-362350d56bf2 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:42.748157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08222824-1f0d-432f-9fda-289514bb2fbd None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:42.780817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e01014-9ade-4bb8-a3da-c697b9826d18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:42.781487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e01014-9ade-4bb8-a3da-c697b9826d18 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.781773 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e01014-9ade-4bb8-a3da-c697b9826d18 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.782067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e01014-9ade-4bb8-a3da-c697b9826d18 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.782914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e01014-9ade-4bb8-a3da-c697b9826d18 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:42.783092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e01014-9ade-4bb8-a3da-c697b9826d18 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:42.784442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e01014-9ade-4bb8-a3da-c697b9826d18 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:42.789828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 507/1008] 10.222.0.25 () {56 vars in 1122 bytes} [Fri Jun 10 12:46:42 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:42.795239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e01014-9ade-4bb8-a3da-c697b9826d18 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:42.856271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08222824-1f0d-432f-9fda-289514bb2fbd None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:42.888246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:42.891227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49608379-2b3d-400b-9c77-ed186ec6049a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:42.891749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e01014-9ade-4bb8-a3da-c697b9826d18 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:42.891872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49608379-2b3d-400b-9c77-ed186ec6049a None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.892164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49608379-2b3d-400b-9c77-ed186ec6049a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.892445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49608379-2b3d-400b-9c77-ed186ec6049a None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.893204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49608379-2b3d-400b-9c77-ed186ec6049a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:42.894587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49608379-2b3d-400b-9c77-ed186ec6049a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:42.924895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69956cab-c7e8-4504-b587-2f7814c04eac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:42.925543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69956cab-c7e8-4504-b587-2f7814c04eac None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.925828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69956cab-c7e8-4504-b587-2f7814c04eac None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.926105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69956cab-c7e8-4504-b587-2f7814c04eac None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.926856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69956cab-c7e8-4504-b587-2f7814c04eac None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:42.928211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69956cab-c7e8-4504-b587-2f7814c04eac None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:42.970102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 502/1009] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:46:42 2022] POST /identity/v3/services => generated 242 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:46:42.971334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b80fbe6-6f43-4142-9dfb-ff7ad0fede35 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.971625 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b80fbe6-6f43-4142-9dfb-ff7ad0fede35 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.971908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b80fbe6-6f43-4142-9dfb-ff7ad0fede35 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.973193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 503/1010] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:42.978859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907ce5a0-272e-433f-a8ad-3547bcb5979f None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:42.979153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907ce5a0-272e-433f-a8ad-3547bcb5979f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:42.979436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907ce5a0-272e-433f-a8ad-3547bcb5979f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:42.999297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-907ce5a0-272e-433f-a8ad-3547bcb5979f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:43.012866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 508/1011] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Jun 10 12:46:42 2022] POST /identity/v3/endpoints => generated 325 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:46:43.013102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-907ce5a0-272e-433f-a8ad-3547bcb5979f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:43.067230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:43.105670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 504/1012] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:42 2022] POST /identity/v3/auth/tokens => generated 1803 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:43.130343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69956cab-c7e8-4504-b587-2f7814c04eac None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:43.238843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69956cab-c7e8-4504-b587-2f7814c04eac None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:43.276049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfb99ba-28a0-4983-be39-ac1a12c689ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:43.277264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfb99ba-28a0-4983-be39-ac1a12c689ce None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:43.277757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfb99ba-28a0-4983-be39-ac1a12c689ce None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:43.278235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfb99ba-28a0-4983-be39-ac1a12c689ce None admin] PATH_INFO: `/v3/services/compute` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:43.279089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfb99ba-28a0-4983-be39-ac1a12c689ce None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:43.280683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfb99ba-28a0-4983-be39-ac1a12c689ce None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-9cfb99ba-28a0-4983-be39-ac1a12c689ce None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:43.292863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:43.293944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jun 10 12:46:43.294804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:43.296844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 509/1013] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:43 2022] GET /identity/v3/services/compute => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:43.300864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907ce5a0-272e-433f-a8ad-3547bcb5979f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:43.407026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907ce5a0-272e-433f-a8ad-3547bcb5979f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:43.440875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b47d2-5e30-4154-8a04-02830abf9c49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:43.441548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b47d2-5e30-4154-8a04-02830abf9c49 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:43.441829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b47d2-5e30-4154-8a04-02830abf9c49 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:43.442634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b47d2-5e30-4154-8a04-02830abf9c49 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:43.443334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b47d2-5e30-4154-8a04-02830abf9c49 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:43.443640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b47d2-5e30-4154-8a04-02830abf9c49 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:43.445044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4b47d2-5e30-4154-8a04-02830abf9c49 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:43.454473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 505/1014] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:43 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:43.456058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfb99ba-28a0-4983-be39-ac1a12c689ce None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:43.483251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:43.563930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfb99ba-28a0-4983-be39-ac1a12c689ce None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:43.599107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-904de4c5-5ec7-4fdc-bb43-220689626a62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:43.599769 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904de4c5-5ec7-4fdc-bb43-220689626a62 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:43.600054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904de4c5-5ec7-4fdc-bb43-220689626a62 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:43.600341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-904de4c5-5ec7-4fdc-bb43-220689626a62 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:43.601076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904de4c5-5ec7-4fdc-bb43-220689626a62 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:43.601389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904de4c5-5ec7-4fdc-bb43-220689626a62 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:43.602761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-904de4c5-5ec7-4fdc-bb43-220689626a62 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:43.612338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 510/1015] 10.222.0.25 () {56 vars in 1112 bytes} [Fri Jun 10 12:46:43 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:43.613614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b47d2-5e30-4154-8a04-02830abf9c49 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:43.649002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:43.716940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b47d2-5e30-4154-8a04-02830abf9c49 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:43.751284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db015d34-1862-45b5-aed9-07c66290e9a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:43.751987 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db015d34-1862-45b5-aed9-07c66290e9a3 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:43.752274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db015d34-1862-45b5-aed9-07c66290e9a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:43.752548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db015d34-1862-45b5-aed9-07c66290e9a3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:43.753321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db015d34-1862-45b5-aed9-07c66290e9a3 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:43.754716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db015d34-1862-45b5-aed9-07c66290e9a3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:43.860876 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 506/1016] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Jun 10 12:46:43 2022] POST /identity/v3/endpoints => generated 334 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:46:45.140755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad0e26-f9fa-43b7-bc86-8fb323b324cd None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.141362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad0e26-f9fa-43b7-bc86-8fb323b324cd None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.141362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ad0e26-f9fa-43b7-bc86-8fb323b324cd None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.142764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 511/1017] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:45.149054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a6869d-1db5-4ea1-bf1e-ee4777992737 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.149872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a6869d-1db5-4ea1-bf1e-ee4777992737 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.149872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a6869d-1db5-4ea1-bf1e-ee4777992737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.169023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-30a6869d-1db5-4ea1-bf1e-ee4777992737 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:45.170245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-30a6869d-1db5-4ea1-bf1e-ee4777992737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:45.204985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b68f75-e70c-4365-9345-a50274469c7e None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.205275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b68f75-e70c-4365-9345-a50274469c7e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.205559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b68f75-e70c-4365-9345-a50274469c7e None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.207196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 512/1018] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:45.212437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace6861f-127a-48f4-9819-d7ffd5d4cafe None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.212731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace6861f-127a-48f4-9819-d7ffd5d4cafe None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.213046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace6861f-127a-48f4-9819-d7ffd5d4cafe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.229465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ace6861f-127a-48f4-9819-d7ffd5d4cafe None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:45.230637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ace6861f-127a-48f4-9819-d7ffd5d4cafe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:45.246026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 507/1019] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:45 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:45.268679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a6869d-1db5-4ea1-bf1e-ee4777992737 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:45.287824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:45.307548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 513/1020] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:45 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:45.331621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace6861f-127a-48f4-9819-d7ffd5d4cafe None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:45.349878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:45.381375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a6869d-1db5-4ea1-bf1e-ee4777992737 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:45.416421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa0c71b-e353-4c63-9816-c08a2758e91a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:45.417167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa0c71b-e353-4c63-9816-c08a2758e91a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.417452 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa0c71b-e353-4c63-9816-c08a2758e91a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.417728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa0c71b-e353-4c63-9816-c08a2758e91a None admin] PATH_INFO: `/v3/services/image` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.418377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa0c71b-e353-4c63-9816-c08a2758e91a None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:45.419757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa0c71b-e353-4c63-9816-c08a2758e91a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-2fa0c71b-e353-4c63-9816-c08a2758e91a None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:45.432281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:45.433451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jun 10 12:46:45.434517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:45.438394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 508/1021] 10.222.0.25 () {56 vars in 1105 bytes} [Fri Jun 10 12:46:45 2022] GET /identity/v3/services/image => generated 86 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:45.439255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa0c71b-e353-4c63-9816-c08a2758e91a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:45.441670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace6861f-127a-48f4-9819-d7ffd5d4cafe None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:45.477991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594b0ed8-d9d2-488a-8f37-6f3a86ed383b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:45.478786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594b0ed8-d9d2-488a-8f37-6f3a86ed383b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.479070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594b0ed8-d9d2-488a-8f37-6f3a86ed383b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.479351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594b0ed8-d9d2-488a-8f37-6f3a86ed383b None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.480526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594b0ed8-d9d2-488a-8f37-6f3a86ed383b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:45.482002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-594b0ed8-d9d2-488a-8f37-6f3a86ed383b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:45.483776 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 514/1022] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:45.489785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594b0ed8-d9d2-488a-8f37-6f3a86ed383b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:45.522378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:45.544397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa0c71b-e353-4c63-9816-c08a2758e91a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:45.580945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44022ec7-a4c6-450c-8ce0-35cdf61d49fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:45.581619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44022ec7-a4c6-450c-8ce0-35cdf61d49fe None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.581905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44022ec7-a4c6-450c-8ce0-35cdf61d49fe None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.582183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44022ec7-a4c6-450c-8ce0-35cdf61d49fe None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.582933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44022ec7-a4c6-450c-8ce0-35cdf61d49fe None admin] RBAC: Adding query filter params (name=image) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:45.583242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44022ec7-a4c6-450c-8ce0-35cdf61d49fe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:45.584063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:45.584615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44022ec7-a4c6-450c-8ce0-35cdf61d49fe None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:45.590157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 509/1023] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:45 2022] GET /identity/v3/services?name=image => generated 122 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:45.595819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44022ec7-a4c6-450c-8ce0-35cdf61d49fe None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:45.606764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594b0ed8-d9d2-488a-8f37-6f3a86ed383b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:45.643458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add8afe5-f10b-428b-b5c5-94f684b55092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:45.644352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add8afe5-f10b-428b-b5c5-94f684b55092 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.644890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add8afe5-f10b-428b-b5c5-94f684b55092 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.645489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add8afe5-f10b-428b-b5c5-94f684b55092 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.646205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add8afe5-f10b-428b-b5c5-94f684b55092 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:45.647644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-add8afe5-f10b-428b-b5c5-94f684b55092 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:45.694378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44022ec7-a4c6-450c-8ce0-35cdf61d49fe None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:45.727406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9807573a-015d-48b1-9ea7-029e97620de8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:45.728065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9807573a-015d-48b1-9ea7-029e97620de8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.728353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9807573a-015d-48b1-9ea7-029e97620de8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.728629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9807573a-015d-48b1-9ea7-029e97620de8 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.729363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9807573a-015d-48b1-9ea7-029e97620de8 None admin] RBAC: Adding query filter params (type=image) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:45.729690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9807573a-015d-48b1-9ea7-029e97620de8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:45.731038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9807573a-015d-48b1-9ea7-029e97620de8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:45.741093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 510/1024] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:45 2022] GET /identity/v3/services?type=image => generated 350 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:45.742205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9807573a-015d-48b1-9ea7-029e97620de8 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:45.744182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 515/1025] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:46:45 2022] POST /identity/v3/projects => generated 311 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:46:45.753796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:45.853720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9807573a-015d-48b1-9ea7-029e97620de8 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:45.891703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247a4c5a-713e-4c7f-ad88-bf0b7a244174 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:45.892382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a4c5a-713e-4c7f-ad88-bf0b7a244174 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:45.892671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a4c5a-713e-4c7f-ad88-bf0b7a244174 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:45.893003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a4c5a-713e-4c7f-ad88-bf0b7a244174 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:45.893818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247a4c5a-713e-4c7f-ad88-bf0b7a244174 None admin] RBAC: Adding query filter params (interface=public, service_id=7c68354212cd4b999a41754e22d0784e, region_id=RegionOne) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:45.894135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247a4c5a-713e-4c7f-ad88-bf0b7a244174 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:45.895550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247a4c5a-713e-4c7f-ad88-bf0b7a244174 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:45.906667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 511/1026] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Jun 10 12:46:45 2022] GET /identity/v3/endpoints?service_id=7c68354212cd4b999a41754e22d0784e&interface=public®ion_id=RegionOne => generated 194 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:45.907718 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add8afe5-f10b-428b-b5c5-94f684b55092 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:46.023523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add8afe5-f10b-428b-b5c5-94f684b55092 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:46.049405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9f52ea-a56e-4d73-822e-02e9453de8c7 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:46.049781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9f52ea-a56e-4d73-822e-02e9453de8c7 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:46.049976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd9f52ea-a56e-4d73-822e-02e9453de8c7 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:46.052034 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 512/1027] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:46.057024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29c6407-fcf2-4b56-9a7b-ef02d97ca64c None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:46.057320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29c6407-fcf2-4b56-9a7b-ef02d97ca64c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:46.057606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a29c6407-fcf2-4b56-9a7b-ef02d97ca64c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:46.059306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402a6b6e-dbb5-4a74-a861-eff7486fae99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:46.059967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402a6b6e-dbb5-4a74-a861-eff7486fae99 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:46.060252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402a6b6e-dbb5-4a74-a861-eff7486fae99 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:46.060532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402a6b6e-dbb5-4a74-a861-eff7486fae99 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:46.061246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402a6b6e-dbb5-4a74-a861-eff7486fae99 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:46.062613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402a6b6e-dbb5-4a74-a861-eff7486fae99 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:46.065361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:46.076808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 516/1028] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Jun 10 12:46:45 2022] GET /identity/v3/services => generated 1957 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 12:46:46.081455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a29c6407-fcf2-4b56-9a7b-ef02d97ca64c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:46.082743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a29c6407-fcf2-4b56-9a7b-ef02d97ca64c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:46.165194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 513/1029] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:46 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:46.189799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402a6b6e-dbb5-4a74-a861-eff7486fae99 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:46.297937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402a6b6e-dbb5-4a74-a861-eff7486fae99 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:46.338110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03a563b-c554-4ab5-97cc-77425b47f528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:46.338926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03a563b-c554-4ab5-97cc-77425b47f528 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:46.339212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03a563b-c554-4ab5-97cc-77425b47f528 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:46.339492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03a563b-c554-4ab5-97cc-77425b47f528 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:46.350604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03a563b-c554-4ab5-97cc-77425b47f528 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-f03a563b-c554-4ab5-97cc-77425b47f528 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:46.358680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:46.359584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:46.360434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:46.361681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 517/1030] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Jun 10 12:46:46 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:46.366379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29c6407-fcf2-4b56-9a7b-ef02d97ca64c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:46.368554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:46.465560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a29c6407-fcf2-4b56-9a7b-ef02d97ca64c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:46.499483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce011dd0-63cd-4716-966e-64fab6f612ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:46.500147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce011dd0-63cd-4716-966e-64fab6f612ae None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:46.500425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce011dd0-63cd-4716-966e-64fab6f612ae None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:46.500698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce011dd0-63cd-4716-966e-64fab6f612ae None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:46.501459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce011dd0-63cd-4716-966e-64fab6f612ae None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:46.501764 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce011dd0-63cd-4716-966e-64fab6f612ae None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:46.503121 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce011dd0-63cd-4716-966e-64fab6f612ae None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:46.516113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 514/1031] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Jun 10 12:46:46 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:46.517726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03a563b-c554-4ab5-97cc-77425b47f528 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:46.580118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:46.621538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03a563b-c554-4ab5-97cc-77425b47f528 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:46.655646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc9892d-28c5-41b5-84d3-538d16f9d0dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:46.656428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc9892d-28c5-41b5-84d3-538d16f9d0dd None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:46.656712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc9892d-28c5-41b5-84d3-538d16f9d0dd None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:46.657021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc9892d-28c5-41b5-84d3-538d16f9d0dd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:46.667335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc9892d-28c5-41b5-84d3-538d16f9d0dd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:46.668796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc9892d-28c5-41b5-84d3-538d16f9d0dd None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-fbc9892d-28c5-41b5-84d3-538d16f9d0dd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:46.685290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:46.686191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:46.687080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:46.687080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:46.687080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:46.687080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:46.687080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 518/1032] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:46.692802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce011dd0-63cd-4716-966e-64fab6f612ae None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:46.756529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:46.794219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce011dd0-63cd-4716-966e-64fab6f612ae None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:46.827372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c0e36b-221b-4bad-8272-e4b8beab82be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:46.828035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c0e36b-221b-4bad-8272-e4b8beab82be None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:46.828321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c0e36b-221b-4bad-8272-e4b8beab82be None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:46.828602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c0e36b-221b-4bad-8272-e4b8beab82be None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:46.829350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c0e36b-221b-4bad-8272-e4b8beab82be None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:46.829660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c0e36b-221b-4bad-8272-e4b8beab82be None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:46.831088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1c0e36b-221b-4bad-8272-e4b8beab82be None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:46.847913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 515/1033] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:46.849925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc9892d-28c5-41b5-84d3-538d16f9d0dd None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:46.951389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc9892d-28c5-41b5-84d3-538d16f9d0dd None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:46.964001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:46.984403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033b86d4-f64f-4662-8e5a-13d92d4cc6f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:46.985219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033b86d4-f64f-4662-8e5a-13d92d4cc6f1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:46.985503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033b86d4-f64f-4662-8e5a-13d92d4cc6f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:46.985779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033b86d4-f64f-4662-8e5a-13d92d4cc6f1 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:46.993710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033b86d4-f64f-4662-8e5a-13d92d4cc6f1 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:46.995154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-033b86d4-f64f-4662-8e5a-13d92d4cc6f1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-033b86d4-f64f-4662-8e5a-13d92d4cc6f1 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:47.009334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:47.010288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:47.011326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 519/1034] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Jun 10 12:46:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:47.016790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c0e36b-221b-4bad-8272-e4b8beab82be None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:47.119330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c0e36b-221b-4bad-8272-e4b8beab82be None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:47.142656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:47.154463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ef34f3-c5a7-4d55-986b-3d6c8f016a04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:47.155134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ef34f3-c5a7-4d55-986b-3d6c8f016a04 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:47.155464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ef34f3-c5a7-4d55-986b-3d6c8f016a04 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:47.155748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ef34f3-c5a7-4d55-986b-3d6c8f016a04 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:47.156574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ef34f3-c5a7-4d55-986b-3d6c8f016a04 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:47.156919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ef34f3-c5a7-4d55-986b-3d6c8f016a04 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:47.158375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ef34f3-c5a7-4d55-986b-3d6c8f016a04 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:47.182547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 516/1035] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Jun 10 12:46:47 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:47.184280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033b86d4-f64f-4662-8e5a-13d92d4cc6f1 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:47.288531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-033b86d4-f64f-4662-8e5a-13d92d4cc6f1 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:47.323530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f15cb73-2972-43fe-abd5-4feb6a43ac45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:47.324178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f15cb73-2972-43fe-abd5-4feb6a43ac45 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:47.324458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f15cb73-2972-43fe-abd5-4feb6a43ac45 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:47.324730 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f15cb73-2972-43fe-abd5-4feb6a43ac45 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:47.367603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:47.393187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f15cb73-2972-43fe-abd5-4feb6a43ac45 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:47.394563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f15cb73-2972-43fe-abd5-4feb6a43ac45 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:47.500474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 520/1036] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:47 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:46:47.507450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ef34f3-c5a7-4d55-986b-3d6c8f016a04 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:47.607748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ef34f3-c5a7-4d55-986b-3d6c8f016a04 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:47.642028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea58826d-2e65-4fe9-96fb-826a6159249d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:47.642812 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea58826d-2e65-4fe9-96fb-826a6159249d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:47.643091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea58826d-2e65-4fe9-96fb-826a6159249d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:47.643411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea58826d-2e65-4fe9-96fb-826a6159249d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:47.653416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea58826d-2e65-4fe9-96fb-826a6159249d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:47.654851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea58826d-2e65-4fe9-96fb-826a6159249d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-ea58826d-2e65-4fe9-96fb-826a6159249d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:47.671900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:47.673147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:47.674078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:47.674078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:47.674078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:47.674078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:47.674839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 517/1037] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:47.680113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f15cb73-2972-43fe-abd5-4feb6a43ac45 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:47.751485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:47.780782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f15cb73-2972-43fe-abd5-4feb6a43ac45 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:47.814800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8130dd9c-d03f-4e56-80d3-2161d17d7ac3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:47.815461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8130dd9c-d03f-4e56-80d3-2161d17d7ac3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:47.815744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8130dd9c-d03f-4e56-80d3-2161d17d7ac3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:47.816018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8130dd9c-d03f-4e56-80d3-2161d17d7ac3 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:47.816774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8130dd9c-d03f-4e56-80d3-2161d17d7ac3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:47.817089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8130dd9c-d03f-4e56-80d3-2161d17d7ac3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:47.818465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8130dd9c-d03f-4e56-80d3-2161d17d7ac3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:47.835323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 521/1038] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:47.836837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea58826d-2e65-4fe9-96fb-826a6159249d None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:47.876624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:47.948479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea58826d-2e65-4fe9-96fb-826a6159249d None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:47.972097 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3907308c-407f-4311-bf03-940256d404fd None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:47.972383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3907308c-407f-4311-bf03-940256d404fd None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:47.972665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3907308c-407f-4311-bf03-940256d404fd None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:47.974116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 522/1039] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:47.979152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a8534-6d97-4b5d-bbe6-1601bd4bd9cd None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:47.979505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a8534-6d97-4b5d-bbe6-1601bd4bd9cd None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:47.979855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9a8534-6d97-4b5d-bbe6-1601bd4bd9cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:47.989374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f645f4-2ac1-4386-bc56-df216e5f35b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:47.990174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f645f4-2ac1-4386-bc56-df216e5f35b1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:47.990460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f645f4-2ac1-4386-bc56-df216e5f35b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:47.990735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f645f4-2ac1-4386-bc56-df216e5f35b1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:47.996635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ab9a8534-6d97-4b5d-bbe6-1601bd4bd9cd None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:47.997638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f645f4-2ac1-4386-bc56-df216e5f35b1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:47.997852 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ab9a8534-6d97-4b5d-bbe6-1601bd4bd9cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:47.999084 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f645f4-2ac1-4386-bc56-df216e5f35b1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-26f645f4-2ac1-4386-bc56-df216e5f35b1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:48.013235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:48.014126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:48.015020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:48.015020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:48.015020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:48.015020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:48.015020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:48.015020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:48.015020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:48.015020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:48.015020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 518/1040] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:48.021876 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f645f4-2ac1-4386-bc56-df216e5f35b1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:48.076924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 523/1041] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:47 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:48.098904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a8534-6d97-4b5d-bbe6-1601bd4bd9cd None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:48.127927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:48.136428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f645f4-2ac1-4386-bc56-df216e5f35b1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:48.165911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:48.174411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d110dd4d-a20f-45bf-a5ad-65759bc80c67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:48.175285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d110dd4d-a20f-45bf-a5ad-65759bc80c67 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:48.175581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d110dd4d-a20f-45bf-a5ad-65759bc80c67 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:48.175858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d110dd4d-a20f-45bf-a5ad-65759bc80c67 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:48.177311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d110dd4d-a20f-45bf-a5ad-65759bc80c67 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:48.177693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d110dd4d-a20f-45bf-a5ad-65759bc80c67 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:48.179511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d110dd4d-a20f-45bf-a5ad-65759bc80c67 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:48.193468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 519/1042] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:48.199405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d110dd4d-a20f-45bf-a5ad-65759bc80c67 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:48.208377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9a8534-6d97-4b5d-bbe6-1601bd4bd9cd None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:48.241111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f8ae05-f3e2-4e89-8c52-2107bf74cf19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:48.241890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f8ae05-f3e2-4e89-8c52-2107bf74cf19 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:48.242169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f8ae05-f3e2-4e89-8c52-2107bf74cf19 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:48.242444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f8ae05-f3e2-4e89-8c52-2107bf74cf19 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:48.243600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f8ae05-f3e2-4e89-8c52-2107bf74cf19 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:48.245046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f8ae05-f3e2-4e89-8c52-2107bf74cf19 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:48.246649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 524/1043] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:48.247787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ce1c54-7ab0-46dd-af9e-cd6f74f13100 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:48.248074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ce1c54-7ab0-46dd-af9e-cd6f74f13100 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:48.248348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ce1c54-7ab0-46dd-af9e-cd6f74f13100 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:48.249575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 525/1044] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:48 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:48.252038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ce1c54-7ab0-46dd-af9e-cd6f74f13100 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:48.299647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d110dd4d-a20f-45bf-a5ad-65759bc80c67 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:48.336217 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732ffdc3-83db-44c7-ab9e-83366fcbd6cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:48.336911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732ffdc3-83db-44c7-ab9e-83366fcbd6cb None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:48.337201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732ffdc3-83db-44c7-ab9e-83366fcbd6cb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:48.337482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732ffdc3-83db-44c7-ab9e-83366fcbd6cb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:48.338311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732ffdc3-83db-44c7-ab9e-83366fcbd6cb None admin] RBAC: Adding query filter params (role.id=ee9ee7e94d4e4ac39a965a63e0abea25, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=e9a0496199134369a0ef740c099eed59) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:48.338626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732ffdc3-83db-44c7-ab9e-83366fcbd6cb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:48.340068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732ffdc3-83db-44c7-ab9e-83366fcbd6cb None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:48.345628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 520/1045] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:48 2022] GET /identity/v3/role_assignments?user.id=e9a0496199134369a0ef740c099eed59&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=ee9ee7e94d4e4ac39a965a63e0abea25 => generated 259 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:48.346793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725d5332-f3ef-4132-8de3-3ce32ce6840e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:48.347083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725d5332-f3ef-4132-8de3-3ce32ce6840e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:48.347363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725d5332-f3ef-4132-8de3-3ce32ce6840e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:48.359084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ce1c54-7ab0-46dd-af9e-cd6f74f13100 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:48.365332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-725d5332-f3ef-4132-8de3-3ce32ce6840e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:48.366580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-725d5332-f3ef-4132-8de3-3ce32ce6840e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:48.381975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:48.396505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:48.403985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e1839d-3329-4a54-979e-e9513a10f0e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:48.404675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e1839d-3329-4a54-979e-e9513a10f0e1 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:48.405013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e1839d-3329-4a54-979e-e9513a10f0e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:48.405371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e1839d-3329-4a54-979e-e9513a10f0e1 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:48.406034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e1839d-3329-4a54-979e-e9513a10f0e1 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:48.407576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e1839d-3329-4a54-979e-e9513a10f0e1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:48.442498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 521/1046] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:48 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:48.465869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725d5332-f3ef-4132-8de3-3ce32ce6840e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:48.500813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 526/1047] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:46:48 2022] POST /identity/v3/users => generated 285 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:46:48.577831 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725d5332-f3ef-4132-8de3-3ce32ce6840e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:48.615682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36184adf-846d-486a-9fd3-d41513ce62f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:48.616483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36184adf-846d-486a-9fd3-d41513ce62f3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:48.616777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36184adf-846d-486a-9fd3-d41513ce62f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:48.617067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36184adf-846d-486a-9fd3-d41513ce62f3 None admin] PATH_INFO: `/v3/services/image` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:48.617724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36184adf-846d-486a-9fd3-d41513ce62f3 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:48.619125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36184adf-846d-486a-9fd3-d41513ce62f3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-36184adf-846d-486a-9fd3-d41513ce62f3 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:48.631829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:48.632943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:48.633897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 522/1048] 10.222.0.25 () {56 vars in 1105 bytes} [Fri Jun 10 12:46:48 2022] GET /identity/v3/services/image => generated 86 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:48.639533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e1839d-3329-4a54-979e-e9513a10f0e1 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:48.753851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e1839d-3329-4a54-979e-e9513a10f0e1 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:48.788280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3129a0-1993-44aa-a4d4-17893d366d3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:48.788973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3129a0-1993-44aa-a4d4-17893d366d3b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:48.789264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3129a0-1993-44aa-a4d4-17893d366d3b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:48.789544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3129a0-1993-44aa-a4d4-17893d366d3b None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:48.790252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3129a0-1993-44aa-a4d4-17893d366d3b None admin] RBAC: Adding query filter params (name=image) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:48.790558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3129a0-1993-44aa-a4d4-17893d366d3b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:48.791938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3129a0-1993-44aa-a4d4-17893d366d3b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:48.797105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 527/1049] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:48 2022] GET /identity/v3/services?name=image => generated 122 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:48.802891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36184adf-846d-486a-9fd3-d41513ce62f3 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:48.820282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:48.905953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36184adf-846d-486a-9fd3-d41513ce62f3 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:48.938287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821df0d3-360f-4010-8b2a-0f90f6906c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:48.938942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821df0d3-360f-4010-8b2a-0f90f6906c35 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:48.939225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821df0d3-360f-4010-8b2a-0f90f6906c35 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:48.939499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821df0d3-360f-4010-8b2a-0f90f6906c35 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:48.940224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821df0d3-360f-4010-8b2a-0f90f6906c35 None admin] RBAC: Adding query filter params (type=image) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:48.940530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821df0d3-360f-4010-8b2a-0f90f6906c35 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:48.942513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821df0d3-360f-4010-8b2a-0f90f6906c35 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:48.947477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 523/1050] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:48 2022] GET /identity/v3/services?type=image => generated 350 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:48.953164 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3129a0-1993-44aa-a4d4-17893d366d3b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:49.005014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:49.056437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3129a0-1993-44aa-a4d4-17893d366d3b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:49.093296 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f6e0220-e747-43e3-a052-004fe9bc29d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:49.094433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e0220-e747-43e3-a052-004fe9bc29d1 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:49.094938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e0220-e747-43e3-a052-004fe9bc29d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:49.095449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f6e0220-e747-43e3-a052-004fe9bc29d1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:49.096891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6e0220-e747-43e3-a052-004fe9bc29d1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:49.099215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f6e0220-e747-43e3-a052-004fe9bc29d1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:49.191222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 528/1051] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Jun 10 12:46:48 2022] POST /identity/v3/endpoints => generated 327 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:46:50.464120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eb2ae2-f72e-4700-8d34-f5bbca7ce5ff None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:50.464673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eb2ae2-f72e-4700-8d34-f5bbca7ce5ff None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:50.464673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1eb2ae2-f72e-4700-8d34-f5bbca7ce5ff None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:50.466115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 524/1052] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:50.471217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe2e39a-f9d3-400f-bba5-d19967b3e353 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:50.471592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe2e39a-f9d3-400f-bba5-d19967b3e353 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:50.471768 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe2e39a-f9d3-400f-bba5-d19967b3e353 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:50.488283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3fe2e39a-f9d3-400f-bba5-d19967b3e353 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:50.490041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3fe2e39a-f9d3-400f-bba5-d19967b3e353 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:50.562799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 529/1053] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:50 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:50.585706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1eb2ae2-f72e-4700-8d34-f5bbca7ce5ff None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:50.669767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:50.677387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ec3ba5-f7e1-4479-ae7e-2e3fa4dad4df None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:50.677683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ec3ba5-f7e1-4479-ae7e-2e3fa4dad4df None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:50.677968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ec3ba5-f7e1-4479-ae7e-2e3fa4dad4df None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:50.679332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 530/1054] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:46:50.684551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5001cf-3a00-4025-a7cd-ddc12c91d5fc None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:50.684880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5001cf-3a00-4025-a7cd-ddc12c91d5fc None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:50.690206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5001cf-3a00-4025-a7cd-ddc12c91d5fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:50.695126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1eb2ae2-f72e-4700-8d34-f5bbca7ce5ff None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:50.702160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-1a5001cf-3a00-4025-a7cd-ddc12c91d5fc None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:50.703347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1a5001cf-3a00-4025-a7cd-ddc12c91d5fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:50.730193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:50.732309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87856beb-d16a-4332-b9d1-5a0681963fe6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:50.733148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87856beb-d16a-4332-b9d1-5a0681963fe6 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:50.733435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87856beb-d16a-4332-b9d1-5a0681963fe6 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:50.733714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87856beb-d16a-4332-b9d1-5a0681963fe6 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:50.744263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87856beb-d16a-4332-b9d1-5a0681963fe6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-87856beb-d16a-4332-b9d1-5a0681963fe6 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:50.752510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:50.753475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:50.754314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 525/1055] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Jun 10 12:46:50 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:50.760807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87856beb-d16a-4332-b9d1-5a0681963fe6 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:50.778118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 531/1056] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:50 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:50.802853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5001cf-3a00-4025-a7cd-ddc12c91d5fc None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:50.866621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87856beb-d16a-4332-b9d1-5a0681963fe6 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:50.902251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5001cf-3a00-4025-a7cd-ddc12c91d5fc None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:50.904029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00da6771-9591-49a1-b8c8-4ea091a88eda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:50.904700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da6771-9591-49a1-b8c8-4ea091a88eda None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:50.905023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da6771-9591-49a1-b8c8-4ea091a88eda None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:50.905423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00da6771-9591-49a1-b8c8-4ea091a88eda None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:50.906167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00da6771-9591-49a1-b8c8-4ea091a88eda None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:50.906472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00da6771-9591-49a1-b8c8-4ea091a88eda None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:50.907852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00da6771-9591-49a1-b8c8-4ea091a88eda None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:50.910952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:50.918369 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 526/1057] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Jun 10 12:46:50 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:50.923800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00da6771-9591-49a1-b8c8-4ea091a88eda None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:50.939792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1617fb-5c5c-4011-aa11-48f86e5c59d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:50.940881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1617fb-5c5c-4011-aa11-48f86e5c59d5 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:50.941344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1617fb-5c5c-4011-aa11-48f86e5c59d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:50.942361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c1617fb-5c5c-4011-aa11-48f86e5c59d5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:50.953947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c1617fb-5c5c-4011-aa11-48f86e5c59d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-5c1617fb-5c5c-4011-aa11-48f86e5c59d5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:50.961812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:50.968738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:50.969614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 532/1058] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:50.972362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1617fb-5c5c-4011-aa11-48f86e5c59d5 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.043118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:51.051112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00da6771-9591-49a1-b8c8-4ea091a88eda None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:51.089720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb563e3-4603-4477-b28e-338d586a2204 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:51.090920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb563e3-4603-4477-b28e-338d586a2204 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.090920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb563e3-4603-4477-b28e-338d586a2204 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.091200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb563e3-4603-4477-b28e-338d586a2204 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.105092 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb563e3-4603-4477-b28e-338d586a2204 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:51.106570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb563e3-4603-4477-b28e-338d586a2204 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:51.120748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c1617fb-5c5c-4011-aa11-48f86e5c59d5 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-ffb563e3-4603-4477-b28e-338d586a2204 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:51.124627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:51.125528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:51.126450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:51.126450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:51.126450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:51.126450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:51.126450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 527/1059] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:51.131830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb563e3-4603-4477-b28e-338d586a2204 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.162779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2f94b0-a38a-4f04-ac81-93ead59dc015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:51.163438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2f94b0-a38a-4f04-ac81-93ead59dc015 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.163721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2f94b0-a38a-4f04-ac81-93ead59dc015 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.163993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2f94b0-a38a-4f04-ac81-93ead59dc015 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.164725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2f94b0-a38a-4f04-ac81-93ead59dc015 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:51.165320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2f94b0-a38a-4f04-ac81-93ead59dc015 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:51.166460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f2f94b0-a38a-4f04-ac81-93ead59dc015 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:51.174995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 533/1060] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:51.181108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2f94b0-a38a-4f04-ac81-93ead59dc015 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.191990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:51.248115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb563e3-4603-4477-b28e-338d586a2204 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:51.267412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:51.285974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f2f94b0-a38a-4f04-ac81-93ead59dc015 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:51.287911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aecc80f-a59e-45b7-8e22-9a8c6855f5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:51.288814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aecc80f-a59e-45b7-8e22-9a8c6855f5c1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.288933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aecc80f-a59e-45b7-8e22-9a8c6855f5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.289611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aecc80f-a59e-45b7-8e22-9a8c6855f5c1 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.289929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aecc80f-a59e-45b7-8e22-9a8c6855f5c1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:51.290234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aecc80f-a59e-45b7-8e22-9a8c6855f5c1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:51.291659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aecc80f-a59e-45b7-8e22-9a8c6855f5c1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:51.310542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 528/1061] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:51.311292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aecc80f-a59e-45b7-8e22-9a8c6855f5c1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.327990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f59b3611-a0b8-4790-893b-21a7c0e08503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:51.328897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b3611-a0b8-4790-893b-21a7c0e08503 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.329180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b3611-a0b8-4790-893b-21a7c0e08503 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.329464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f59b3611-a0b8-4790-893b-21a7c0e08503 None admin] PATH_INFO: `/v3/users/9af76e657f2744aab4685410cbc75d70` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.346118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59b3611-a0b8-4790-893b-21a7c0e08503 None admin] RBAC: Authorizing `identity:get_user(user_id=9af76e657f2744aab4685410cbc75d70)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:51.347567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f59b3611-a0b8-4790-893b-21a7c0e08503 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:51.349878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 534/1062] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:46:51 2022] GET /identity/v3/users/9af76e657f2744aab4685410cbc75d70 => generated 285 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:51.351133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbecafc3-6c01-4d5a-96a4-71ce32d4388f None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.351387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbecafc3-6c01-4d5a-96a4-71ce32d4388f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.351666 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbecafc3-6c01-4d5a-96a4-71ce32d4388f None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.353038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 535/1063] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:51 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:51.355912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbecafc3-6c01-4d5a-96a4-71ce32d4388f None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.419986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aecc80f-a59e-45b7-8e22-9a8c6855f5c1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:51.430052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:51.453629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21ffff2-a4bc-4e04-a6fa-fa3752c6e37f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:51.454427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21ffff2-a4bc-4e04-a6fa-fa3752c6e37f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.454673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbecafc3-6c01-4d5a-96a4-71ce32d4388f None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:51.454772 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21ffff2-a4bc-4e04-a6fa-fa3752c6e37f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.454974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21ffff2-a4bc-4e04-a6fa-fa3752c6e37f None admin] PATH_INFO: `/v3/users/nova` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.463963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21ffff2-a4bc-4e04-a6fa-fa3752c6e37f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:51.465455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21ffff2-a4bc-4e04-a6fa-fa3752c6e37f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-f21ffff2-a4bc-4e04-a6fa-fa3752c6e37f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:51.479950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:51.481007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:51.481941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 529/1064] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Jun 10 12:46:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:51.482550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e93edc-fad9-42b1-80d2-02d00e59b9a8 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.482619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e93edc-fad9-42b1-80d2-02d00e59b9a8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.482901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e93edc-fad9-42b1-80d2-02d00e59b9a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.497852 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb3bc33-996c-4f7d-bfa3-5cca73db0017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:51.498501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb3bc33-996c-4f7d-bfa3-5cca73db0017 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.498785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb3bc33-996c-4f7d-bfa3-5cca73db0017 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.499061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb3bc33-996c-4f7d-bfa3-5cca73db0017 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.500552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e8e93edc-fad9-42b1-80d2-02d00e59b9a8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:51.501771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e8e93edc-fad9-42b1-80d2-02d00e59b9a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:51.511829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:51.567584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb3bc33-996c-4f7d-bfa3-5cca73db0017 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:51.568481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 530/1065] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:51 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:51.569070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cb3bc33-996c-4f7d-bfa3-5cca73db0017 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:51.569359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e93edc-fad9-42b1-80d2-02d00e59b9a8 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.667174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:51.672986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 536/1066] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:51 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jun 10 12:46:51.672986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb3bc33-996c-4f7d-bfa3-5cca73db0017 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.675175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e93edc-fad9-42b1-80d2-02d00e59b9a8 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:51.708536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410baa97-1f25-4556-8f8e-1a8cc08faa2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:51.709235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410baa97-1f25-4556-8f8e-1a8cc08faa2f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.709530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410baa97-1f25-4556-8f8e-1a8cc08faa2f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.709668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:51.709810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410baa97-1f25-4556-8f8e-1a8cc08faa2f None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.710650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410baa97-1f25-4556-8f8e-1a8cc08faa2f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:51.710958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410baa97-1f25-4556-8f8e-1a8cc08faa2f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:51.712391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410baa97-1f25-4556-8f8e-1a8cc08faa2f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:51.730142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 531/1067] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Jun 10 12:46:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:51.731041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410baa97-1f25-4556-8f8e-1a8cc08faa2f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.765510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb3bc33-996c-4f7d-bfa3-5cca73db0017 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:51.798393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5605004c-f506-4024-aef2-5fe61dddd749 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:51.799036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5605004c-f506-4024-aef2-5fe61dddd749 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.799322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5605004c-f506-4024-aef2-5fe61dddd749 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.799599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5605004c-f506-4024-aef2-5fe61dddd749 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.829416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410baa97-1f25-4556-8f8e-1a8cc08faa2f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:51.874815 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8842bc14-da7d-4d4f-9c49-8f292c7824c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:51.875674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8842bc14-da7d-4d4f-9c49-8f292c7824c5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:51.875957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8842bc14-da7d-4d4f-9c49-8f292c7824c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:51.876173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5605004c-f506-4024-aef2-5fe61dddd749 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:51.876283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8842bc14-da7d-4d4f-9c49-8f292c7824c5 None admin] PATH_INFO: `/v3/projects/ac14f093e9334a2188fde0cc97c57fc4` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:51.877405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8842bc14-da7d-4d4f-9c49-8f292c7824c5 None admin] RBAC: Authorizing `identity:get_project(project_id=ac14f093e9334a2188fde0cc97c57fc4)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:51.877664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5605004c-f506-4024-aef2-5fe61dddd749 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:51.878824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8842bc14-da7d-4d4f-9c49-8f292c7824c5 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:51.880586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 532/1068] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:46:51 2022] GET /identity/v3/projects/ac14f093e9334a2188fde0cc97c57fc4 => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:51.881371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8842bc14-da7d-4d4f-9c49-8f292c7824c5 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.912289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:51.919178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:51.976827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 537/1069] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:51 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:46:51.976827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5605004c-f506-4024-aef2-5fe61dddd749 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:51.984020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8842bc14-da7d-4d4f-9c49-8f292c7824c5 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:52.017342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af44f1a8-4c42-4c3e-981f-304463bdd898 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:52.018224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af44f1a8-4c42-4c3e-981f-304463bdd898 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:52.018511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af44f1a8-4c42-4c3e-981f-304463bdd898 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:52.018790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af44f1a8-4c42-4c3e-981f-304463bdd898 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:52.029418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af44f1a8-4c42-4c3e-981f-304463bdd898 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:52.030873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af44f1a8-4c42-4c3e-981f-304463bdd898 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-af44f1a8-4c42-4c3e-981f-304463bdd898 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:52.048662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:52.049745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:52.050636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:52.050636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:52.050636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:52.050636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:52.050636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 533/1070] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:52.051057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af44f1a8-4c42-4c3e-981f-304463bdd898 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:52.077326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5605004c-f506-4024-aef2-5fe61dddd749 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:52.110594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b9a67-97b6-4e32-b9d0-0b0162460f2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:52.111243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b9a67-97b6-4e32-b9d0-0b0162460f2e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:52.111531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b9a67-97b6-4e32-b9d0-0b0162460f2e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:52.111816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7b9a67-97b6-4e32-b9d0-0b0162460f2e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:52.112641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7b9a67-97b6-4e32-b9d0-0b0162460f2e None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=ac14f093e9334a2188fde0cc97c57fc4, user.id=9af76e657f2744aab4685410cbc75d70) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:52.112989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7b9a67-97b6-4e32-b9d0-0b0162460f2e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:52.114956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7b9a67-97b6-4e32-b9d0-0b0162460f2e None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:52.120317 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 538/1071] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:51 2022] GET /identity/v3/role_assignments?user.id=9af76e657f2744aab4685410cbc75d70&scope.project.id=ac14f093e9334a2188fde0cc97c57fc4&role.id=766799d1b6c049168590571f1a959b3b => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:52.121122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b9a67-97b6-4e32-b9d0-0b0162460f2e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:52.132310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:52.137934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:52.156578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af44f1a8-4c42-4c3e-981f-304463bdd898 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:52.196313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05af7bad-1b5e-4a4d-baf0-71e000f6e338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:52.197162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05af7bad-1b5e-4a4d-baf0-71e000f6e338 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:52.197447 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05af7bad-1b5e-4a4d-baf0-71e000f6e338 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:52.197724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05af7bad-1b5e-4a4d-baf0-71e000f6e338 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:52.209134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05af7bad-1b5e-4a4d-baf0-71e000f6e338 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:52.210635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05af7bad-1b5e-4a4d-baf0-71e000f6e338 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-05af7bad-1b5e-4a4d-baf0-71e000f6e338 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:52.230248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:52.231355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:52.237480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:52.237480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:52.237480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:52.237480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:52.237480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 534/1072] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:52.237480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05af7bad-1b5e-4a4d-baf0-71e000f6e338 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:52.251325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7b9a67-97b6-4e32-b9d0-0b0162460f2e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:52.290815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc785fd8-b906-486e-8dc2-c774b66af3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:52.291483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc785fd8-b906-486e-8dc2-c774b66af3e4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:52.291770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc785fd8-b906-486e-8dc2-c774b66af3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:52.292048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc785fd8-b906-486e-8dc2-c774b66af3e4 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:52.292798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc785fd8-b906-486e-8dc2-c774b66af3e4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:52.293119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc785fd8-b906-486e-8dc2-c774b66af3e4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:52.294506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc785fd8-b906-486e-8dc2-c774b66af3e4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:52.317513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 539/1073] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:52.318734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc785fd8-b906-486e-8dc2-c774b66af3e4 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:52.352884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05af7bad-1b5e-4a4d-baf0-71e000f6e338 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:52.384494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:52.390184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:52.390768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a844d2-ce6f-4fc6-becb-7f7692bb4f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:52.391476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a844d2-ce6f-4fc6-becb-7f7692bb4f7b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:52.391720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a844d2-ce6f-4fc6-becb-7f7692bb4f7b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:52.392001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a844d2-ce6f-4fc6-becb-7f7692bb4f7b None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:52.392822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a844d2-ce6f-4fc6-becb-7f7692bb4f7b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:52.393115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a844d2-ce6f-4fc6-becb-7f7692bb4f7b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:52.395354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a844d2-ce6f-4fc6-becb-7f7692bb4f7b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:52.406682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 535/1074] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:52.432705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc785fd8-b906-486e-8dc2-c774b66af3e4 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:52.471526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c6bfc-3b1d-4c83-ac58-6a719e3ad0c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:52.472345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c6bfc-3b1d-4c83-ac58-6a719e3ad0c8 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:52.472670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c6bfc-3b1d-4c83-ac58-6a719e3ad0c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:52.472986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3c6bfc-3b1d-4c83-ac58-6a719e3ad0c8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:52.480820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c6bfc-3b1d-4c83-ac58-6a719e3ad0c8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:52.482217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3c6bfc-3b1d-4c83-ac58-6a719e3ad0c8 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-ea3c6bfc-3b1d-4c83-ac58-6a719e3ad0c8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:52.497598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:52.499032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:52.499947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:52.499947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:52.499947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:52.499947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:52.499947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:52.499947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:52.499947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:52.499947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:52.499947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 540/1075] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:52.506325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a844d2-ce6f-4fc6-becb-7f7692bb4f7b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:52.619350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a844d2-ce6f-4fc6-becb-7f7692bb4f7b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:52.655307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c1a5b93-0dbe-4a1a-bf3d-a6307686af67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:52.656005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1a5b93-0dbe-4a1a-bf3d-a6307686af67 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:52.656295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1a5b93-0dbe-4a1a-bf3d-a6307686af67 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:52.656576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c1a5b93-0dbe-4a1a-bf3d-a6307686af67 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:52.657403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1a5b93-0dbe-4a1a-bf3d-a6307686af67 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:52.657714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1a5b93-0dbe-4a1a-bf3d-a6307686af67 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:52.659157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c1a5b93-0dbe-4a1a-bf3d-a6307686af67 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:52.672684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 536/1076] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:52.678672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c6bfc-3b1d-4c83-ac58-6a719e3ad0c8 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:52.780539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3c6bfc-3b1d-4c83-ac58-6a719e3ad0c8 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:52.805527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:52.813611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368910d7-9e38-42d7-a945-8c27ee2272fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:52.814495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368910d7-9e38-42d7-a945-8c27ee2272fe None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:52.818842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368910d7-9e38-42d7-a945-8c27ee2272fe None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:52.818842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368910d7-9e38-42d7-a945-8c27ee2272fe None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/e9a0496199134369a0ef740c099eed59/roles/ee9ee7e94d4e4ac39a965a63e0abea25` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:52.818842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368910d7-9e38-42d7-a945-8c27ee2272fe None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=e9a0496199134369a0ef740c099eed59, role_id=ee9ee7e94d4e4ac39a965a63e0abea25)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:52.819257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368910d7-9e38-42d7-a945-8c27ee2272fe None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:52.895427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 541/1077] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:46:52 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/e9a0496199134369a0ef740c099eed59/roles/ee9ee7e94d4e4ac39a965a63e0abea25 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:46:54.222509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96442580-3cde-41f5-a7dd-c46333abd6f3 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:54.223033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96442580-3cde-41f5-a7dd-c46333abd6f3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:54.223124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96442580-3cde-41f5-a7dd-c46333abd6f3 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:54.224452 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 537/1078] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:54.229694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b732ec1-64dd-4e28-aad6-ec84a6f8f995 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:54.230128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b732ec1-64dd-4e28-aad6-ec84a6f8f995 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:54.230265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b732ec1-64dd-4e28-aad6-ec84a6f8f995 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:54.246919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-8b732ec1-64dd-4e28-aad6-ec84a6f8f995 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:54.248084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8b732ec1-64dd-4e28-aad6-ec84a6f8f995 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:54.311472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 542/1079] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:54 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:54.334071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96442580-3cde-41f5-a7dd-c46333abd6f3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:54.390041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:54.444692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96442580-3cde-41f5-a7dd-c46333abd6f3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:54.476582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de9627d-b526-4a31-aeb3-fb15aaaedede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:54.477327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de9627d-b526-4a31-aeb3-fb15aaaedede None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:54.477607 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de9627d-b526-4a31-aeb3-fb15aaaedede None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:54.477880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de9627d-b526-4a31-aeb3-fb15aaaedede None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:54.487566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de9627d-b526-4a31-aeb3-fb15aaaedede None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-8de9627d-b526-4a31-aeb3-fb15aaaedede None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:54.495466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:54.496401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:54.497687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 538/1080] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:54.502682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b732ec1-64dd-4e28-aad6-ec84a6f8f995 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:54.602199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b732ec1-64dd-4e28-aad6-ec84a6f8f995 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:54.618890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:54.635535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bbd8a2-087e-4514-96ab-de6c2b3a9338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:54.636196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bbd8a2-087e-4514-96ab-de6c2b3a9338 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:54.636476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bbd8a2-087e-4514-96ab-de6c2b3a9338 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:54.636752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89bbd8a2-087e-4514-96ab-de6c2b3a9338 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:54.637509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bbd8a2-087e-4514-96ab-de6c2b3a9338 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:54.637813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bbd8a2-087e-4514-96ab-de6c2b3a9338 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:54.639171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89bbd8a2-087e-4514-96ab-de6c2b3a9338 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:54.647642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 543/1081] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:54.653691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de9627d-b526-4a31-aeb3-fb15aaaedede None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:54.772467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de9627d-b526-4a31-aeb3-fb15aaaedede None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:54.782741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:54.805386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63b9562-d120-4e03-8189-b440f1fc461b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:54.806250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63b9562-d120-4e03-8189-b440f1fc461b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:54.806546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63b9562-d120-4e03-8189-b440f1fc461b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:54.806811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63b9562-d120-4e03-8189-b440f1fc461b None admin] PATH_INFO: `/v3/users/9af76e657f2744aab4685410cbc75d70` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:54.807982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63b9562-d120-4e03-8189-b440f1fc461b None admin] RBAC: Authorizing `identity:get_user(user_id=9af76e657f2744aab4685410cbc75d70)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:54.809462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63b9562-d120-4e03-8189-b440f1fc461b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:54.811088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 539/1082] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:46:54 2022] GET /identity/v3/users/9af76e657f2744aab4685410cbc75d70 => generated 285 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:54.817228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bbd8a2-087e-4514-96ab-de6c2b3a9338 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:54.914535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89bbd8a2-087e-4514-96ab-de6c2b3a9338 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:54.951619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583f47ca-cb64-4c93-a849-cdaa2f4cba6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:54.952475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583f47ca-cb64-4c93-a849-cdaa2f4cba6b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:54.952755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583f47ca-cb64-4c93-a849-cdaa2f4cba6b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:54.953054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583f47ca-cb64-4c93-a849-cdaa2f4cba6b None admin] PATH_INFO: `/v3/projects/ac14f093e9334a2188fde0cc97c57fc4` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:54.954230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583f47ca-cb64-4c93-a849-cdaa2f4cba6b None admin] RBAC: Authorizing `identity:get_project(project_id=ac14f093e9334a2188fde0cc97c57fc4)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:54.955682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583f47ca-cb64-4c93-a849-cdaa2f4cba6b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:54.957982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 544/1083] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:46:54 2022] GET /identity/v3/projects/ac14f093e9334a2188fde0cc97c57fc4 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:54.963982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63b9562-d120-4e03-8189-b440f1fc461b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:55.033846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9f5f65-858e-4039-a6f9-802749ac7c7c None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:55.034058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9f5f65-858e-4039-a6f9-802749ac7c7c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:55.034342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9f5f65-858e-4039-a6f9-802749ac7c7c None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:55.035750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 545/1084] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:55.040613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2856c8d-86d2-4423-b695-55a7392ca27d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:55.040942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2856c8d-86d2-4423-b695-55a7392ca27d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:55.041227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2856c8d-86d2-4423-b695-55a7392ca27d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:55.059747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e2856c8d-86d2-4423-b695-55a7392ca27d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:55.061830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e2856c8d-86d2-4423-b695-55a7392ca27d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:55.070050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63b9562-d120-4e03-8189-b440f1fc461b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:55.102363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:55.103238 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c159f3c-7253-4631-8ee6-baeed3b7c344 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:55.104117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c159f3c-7253-4631-8ee6-baeed3b7c344 None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:55.104394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c159f3c-7253-4631-8ee6-baeed3b7c344 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:55.104667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c159f3c-7253-4631-8ee6-baeed3b7c344 None admin] PATH_INFO: `/v3/projects/ac14f093e9334a2188fde0cc97c57fc4/users/9af76e657f2744aab4685410cbc75d70/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:55.106826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c159f3c-7253-4631-8ee6-baeed3b7c344 None admin] RBAC: Authorizing `identity:create_grant(project_id=ac14f093e9334a2188fde0cc97c57fc4, user_id=9af76e657f2744aab4685410cbc75d70, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:55.108256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c159f3c-7253-4631-8ee6-baeed3b7c344 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:55.126327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 546/1085] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:55 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:46:55.150463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2856c8d-86d2-4423-b695-55a7392ca27d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:55.215184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 540/1086] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:46:54 2022] PUT /identity/v3/projects/ac14f093e9334a2188fde0cc97c57fc4/users/9af76e657f2744aab4685410cbc75d70/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:46:55.248128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2856c8d-86d2-4423-b695-55a7392ca27d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:55.284449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d926e4f1-2287-430c-a5ed-004defc35889 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:55.285245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d926e4f1-2287-430c-a5ed-004defc35889 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:55.285526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d926e4f1-2287-430c-a5ed-004defc35889 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:55.285802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d926e4f1-2287-430c-a5ed-004defc35889 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:55.297103 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d926e4f1-2287-430c-a5ed-004defc35889 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-d926e4f1-2287-430c-a5ed-004defc35889 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:55.305984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:55.307142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:55.308168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 547/1087] 10.222.0.25 () {56 vars in 1120 bytes} [Fri Jun 10 12:46:55 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:46:55.313404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c159f3c-7253-4631-8ee6-baeed3b7c344 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:55.356449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:55.420448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c159f3c-7253-4631-8ee6-baeed3b7c344 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:55.452938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b035b07-0614-44bc-a631-3b9d86c37bb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:55.453607 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b035b07-0614-44bc-a631-3b9d86c37bb5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:55.453888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b035b07-0614-44bc-a631-3b9d86c37bb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:55.454166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b035b07-0614-44bc-a631-3b9d86c37bb5 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:55.454903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b035b07-0614-44bc-a631-3b9d86c37bb5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:55.455209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b035b07-0614-44bc-a631-3b9d86c37bb5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:55.456573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b035b07-0614-44bc-a631-3b9d86c37bb5 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:55.464917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 541/1088] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Jun 10 12:46:55 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:55.470504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d926e4f1-2287-430c-a5ed-004defc35889 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:55.504433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:55.565922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d926e4f1-2287-430c-a5ed-004defc35889 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:55.599132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829a4c8b-6b38-4cd9-b39e-2b2cfc3dbbbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:55.599906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829a4c8b-6b38-4cd9-b39e-2b2cfc3dbbbe None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:55.600184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829a4c8b-6b38-4cd9-b39e-2b2cfc3dbbbe None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:55.600455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829a4c8b-6b38-4cd9-b39e-2b2cfc3dbbbe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:55.610035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829a4c8b-6b38-4cd9-b39e-2b2cfc3dbbbe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:55.611862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829a4c8b-6b38-4cd9-b39e-2b2cfc3dbbbe None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-829a4c8b-6b38-4cd9-b39e-2b2cfc3dbbbe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:55.630606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:55.631695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:55.633109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:55.633109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:55.633109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:55.633109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:55.633109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 548/1089] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:55.637468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b035b07-0614-44bc-a631-3b9d86c37bb5 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:55.733412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:55.736567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b035b07-0614-44bc-a631-3b9d86c37bb5 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:55.768274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a12884-7175-4c46-bf10-98ffb46f3348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:55.768957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a12884-7175-4c46-bf10-98ffb46f3348 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:55.769749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a12884-7175-4c46-bf10-98ffb46f3348 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:55.770044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a12884-7175-4c46-bf10-98ffb46f3348 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:55.770765 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a12884-7175-4c46-bf10-98ffb46f3348 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:55.771072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a12884-7175-4c46-bf10-98ffb46f3348 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:55.772444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a12884-7175-4c46-bf10-98ffb46f3348 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:55.784693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 542/1090] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:55.790807 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829a4c8b-6b38-4cd9-b39e-2b2cfc3dbbbe None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:55.873999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:55.885183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829a4c8b-6b38-4cd9-b39e-2b2cfc3dbbbe None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:55.919368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23bed58-6cbe-4250-abfb-8b96aa6a412a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:55.920151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23bed58-6cbe-4250-abfb-8b96aa6a412a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:55.920430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23bed58-6cbe-4250-abfb-8b96aa6a412a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:55.920704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c23bed58-6cbe-4250-abfb-8b96aa6a412a None admin] PATH_INFO: `/v3/users/nova` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:55.927536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23bed58-6cbe-4250-abfb-8b96aa6a412a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:55.929002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c23bed58-6cbe-4250-abfb-8b96aa6a412a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c23bed58-6cbe-4250-abfb-8b96aa6a412a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:46:55.942755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:55.943876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jun 10 12:46:55.945054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:55.948794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 549/1091] 10.222.0.25 () {56 vars in 1093 bytes} [Fri Jun 10 12:46:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:55.949596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a12884-7175-4c46-bf10-98ffb46f3348 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:56.059838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a12884-7175-4c46-bf10-98ffb46f3348 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:56.102470 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097f10e3-7314-4176-b5ff-a33dbb909f7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:56.103134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097f10e3-7314-4176-b5ff-a33dbb909f7f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:56.103416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097f10e3-7314-4176-b5ff-a33dbb909f7f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:56.103686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097f10e3-7314-4176-b5ff-a33dbb909f7f None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:56.104513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097f10e3-7314-4176-b5ff-a33dbb909f7f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:56.104848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097f10e3-7314-4176-b5ff-a33dbb909f7f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:56.106272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097f10e3-7314-4176-b5ff-a33dbb909f7f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:56.128137 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 543/1092] 10.222.0.25 () {56 vars in 1133 bytes} [Fri Jun 10 12:46:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:56.129230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23bed58-6cbe-4250-abfb-8b96aa6a412a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:56.224916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c23bed58-6cbe-4250-abfb-8b96aa6a412a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:56.257275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242d878-5527-4c07-9bee-ac12f92edb91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:56.257907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242d878-5527-4c07-9bee-ac12f92edb91 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:56.258189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242d878-5527-4c07-9bee-ac12f92edb91 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:56.258464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242d878-5527-4c07-9bee-ac12f92edb91 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:56.275222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:56.326626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242d878-5527-4c07-9bee-ac12f92edb91 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:56.328097 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242d878-5527-4c07-9bee-ac12f92edb91 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:56.428634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 550/1093] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:56 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jun 10 12:46:56.434455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097f10e3-7314-4176-b5ff-a33dbb909f7f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:56.453202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:56.532020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097f10e3-7314-4176-b5ff-a33dbb909f7f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:56.564733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4321f954-4d31-44ac-a3b7-fb19c2569991 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:56.565541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4321f954-4d31-44ac-a3b7-fb19c2569991 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:56.565823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4321f954-4d31-44ac-a3b7-fb19c2569991 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:56.566645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4321f954-4d31-44ac-a3b7-fb19c2569991 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:56.576281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4321f954-4d31-44ac-a3b7-fb19c2569991 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:56.577755 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4321f954-4d31-44ac-a3b7-fb19c2569991 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-4321f954-4d31-44ac-a3b7-fb19c2569991 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:46:56.594574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:56.595480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:56.596364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:46:56.596364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:46:56.596364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:46:56.596364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:56.596364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 544/1094] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:46:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:56.601766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242d878-5527-4c07-9bee-ac12f92edb91 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:56.649406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:56.697688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242d878-5527-4c07-9bee-ac12f92edb91 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:56.729084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0200b45-6769-4cbc-ac49-473b514e9323 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:56.729736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0200b45-6769-4cbc-ac49-473b514e9323 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:56.730015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0200b45-6769-4cbc-ac49-473b514e9323 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:56.730290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0200b45-6769-4cbc-ac49-473b514e9323 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:56.730998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0200b45-6769-4cbc-ac49-473b514e9323 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:56.731300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0200b45-6769-4cbc-ac49-473b514e9323 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:56.732660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0200b45-6769-4cbc-ac49-473b514e9323 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:56.744499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 551/1095] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:46:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:56.750219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4321f954-4d31-44ac-a3b7-fb19c2569991 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:56.818282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:56.847233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4321f954-4d31-44ac-a3b7-fb19c2569991 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:56.880940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dab059d-f894-44c4-b87c-e686ad1d40b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:56.881719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dab059d-f894-44c4-b87c-e686ad1d40b3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:56.881997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dab059d-f894-44c4-b87c-e686ad1d40b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:56.882270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dab059d-f894-44c4-b87c-e686ad1d40b3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:56.888708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dab059d-f894-44c4-b87c-e686ad1d40b3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:56.890160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dab059d-f894-44c4-b87c-e686ad1d40b3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-9dab059d-f894-44c4-b87c-e686ad1d40b3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:46:56.903776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:56.904709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:56.905656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:46:56.905656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:46:56.905656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:46:56.905656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:46:56.905656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:46:56.905656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:46:56.905656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:46:56.905656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:46:56.905656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 545/1096] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:46:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:56.911280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0200b45-6769-4cbc-ac49-473b514e9323 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:57.006122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0200b45-6769-4cbc-ac49-473b514e9323 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:57.014822 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:57.038634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16730c2-7e03-4fda-9a13-be181c95afd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:57.039281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16730c2-7e03-4fda-9a13-be181c95afd6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:57.039566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16730c2-7e03-4fda-9a13-be181c95afd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:57.039842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16730c2-7e03-4fda-9a13-be181c95afd6 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:57.040677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16730c2-7e03-4fda-9a13-be181c95afd6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:57.041009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16730c2-7e03-4fda-9a13-be181c95afd6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:57.042436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16730c2-7e03-4fda-9a13-be181c95afd6 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:57.055066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 552/1097] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:46:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:57.061211 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dab059d-f894-44c4-b87c-e686ad1d40b3 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:57.163192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dab059d-f894-44c4-b87c-e686ad1d40b3 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:57.189909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:57.196194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a773c04-3462-4046-b877-930e70b335ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:57.196882 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a773c04-3462-4046-b877-930e70b335ae None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:57.197168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a773c04-3462-4046-b877-930e70b335ae None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:57.197444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a773c04-3462-4046-b877-930e70b335ae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:57.198270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a773c04-3462-4046-b877-930e70b335ae None admin] RBAC: Adding query filter params (role.id=ee9ee7e94d4e4ac39a965a63e0abea25, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=e9a0496199134369a0ef740c099eed59) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:57.198584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a773c04-3462-4046-b877-930e70b335ae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:57.200022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a773c04-3462-4046-b877-930e70b335ae None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:57.205752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 546/1098] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:57 2022] GET /identity/v3/role_assignments?user.id=e9a0496199134369a0ef740c099eed59&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=ee9ee7e94d4e4ac39a965a63e0abea25 => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:46:57.336965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88afefa-b9ca-4bf7-9271-f895c9369e80 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:57.337353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88afefa-b9ca-4bf7-9271-f895c9369e80 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:57.337516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88afefa-b9ca-4bf7-9271-f895c9369e80 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:57.338889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 553/1099] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:46:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:46:57.344441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f81586a-3d2e-4cd4-a39e-4c1fb675823b None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:57.344738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f81586a-3d2e-4cd4-a39e-4c1fb675823b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:57.345056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f81586a-3d2e-4cd4-a39e-4c1fb675823b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:57.362776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3f81586a-3d2e-4cd4-a39e-4c1fb675823b None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:46:57.364007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3f81586a-3d2e-4cd4-a39e-4c1fb675823b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:46:57.431660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 547/1100] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:46:57 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:46:57.454564 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88afefa-b9ca-4bf7-9271-f895c9369e80 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:57.555754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88afefa-b9ca-4bf7-9271-f895c9369e80 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:57.589753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8eb91e-9fd8-465c-a529-44d1b0509d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:57.590468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8eb91e-9fd8-465c-a529-44d1b0509d17 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:57.590744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8eb91e-9fd8-465c-a529-44d1b0509d17 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:57.591016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e8eb91e-9fd8-465c-a529-44d1b0509d17 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:57.601526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e8eb91e-9fd8-465c-a529-44d1b0509d17 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-6e8eb91e-9fd8-465c-a529-44d1b0509d17 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:46:57.609033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:46:57.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:46:57.610916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 554/1101] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:46:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:46:57.616257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f81586a-3d2e-4cd4-a39e-4c1fb675823b None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:57.717135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f81586a-3d2e-4cd4-a39e-4c1fb675823b None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:57.740614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:57.750967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b50194-7450-42b7-8fc6-3200f6aeae0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:57.751641 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b50194-7450-42b7-8fc6-3200f6aeae0b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:57.751923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b50194-7450-42b7-8fc6-3200f6aeae0b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:57.752203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b50194-7450-42b7-8fc6-3200f6aeae0b None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:57.752985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b50194-7450-42b7-8fc6-3200f6aeae0b None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:57.762904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b50194-7450-42b7-8fc6-3200f6aeae0b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:57.762904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b50194-7450-42b7-8fc6-3200f6aeae0b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:57.763538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 548/1102] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:46:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:57.769804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8eb91e-9fd8-465c-a529-44d1b0509d17 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:57.800008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:57.873292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e8eb91e-9fd8-465c-a529-44d1b0509d17 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:57.908076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28514a0-b65f-4fde-a420-d9068527a794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:57.908984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28514a0-b65f-4fde-a420-d9068527a794 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:57.910200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28514a0-b65f-4fde-a420-d9068527a794 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:57.910200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28514a0-b65f-4fde-a420-d9068527a794 None admin] PATH_INFO: `/v3/users/9af76e657f2744aab4685410cbc75d70` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:57.910788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28514a0-b65f-4fde-a420-d9068527a794 None admin] RBAC: Authorizing `identity:get_user(user_id=9af76e657f2744aab4685410cbc75d70)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:57.912248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28514a0-b65f-4fde-a420-d9068527a794 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:57.913958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 555/1103] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:46:57 2022] GET /identity/v3/users/9af76e657f2744aab4685410cbc75d70 => generated 285 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:57.919874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b50194-7450-42b7-8fc6-3200f6aeae0b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:58.029634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b50194-7450-42b7-8fc6-3200f6aeae0b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:58.064033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5b8ff1-8acc-4274-8589-279cb233196a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:58.064677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5b8ff1-8acc-4274-8589-279cb233196a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:58.065043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5b8ff1-8acc-4274-8589-279cb233196a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:58.065328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5b8ff1-8acc-4274-8589-279cb233196a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:58.123997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:58.136337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5b8ff1-8acc-4274-8589-279cb233196a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:58.137953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5b8ff1-8acc-4274-8589-279cb233196a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:58.231535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 549/1104] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:46:57 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:46:58.237481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28514a0-b65f-4fde-a420-d9068527a794 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:58.326882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:58.332809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28514a0-b65f-4fde-a420-d9068527a794 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:58.365208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f93876-12d0-4de8-9ac0-ccff019fa1f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:58.366087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f93876-12d0-4de8-9ac0-ccff019fa1f0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:58.366370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f93876-12d0-4de8-9ac0-ccff019fa1f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:58.366649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f93876-12d0-4de8-9ac0-ccff019fa1f0 None admin] PATH_INFO: `/v3/projects/ac14f093e9334a2188fde0cc97c57fc4` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:58.367812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f93876-12d0-4de8-9ac0-ccff019fa1f0 None admin] RBAC: Authorizing `identity:get_project(project_id=ac14f093e9334a2188fde0cc97c57fc4)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:58.369274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f93876-12d0-4de8-9ac0-ccff019fa1f0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:58.370951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 556/1105] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:46:58 2022] GET /identity/v3/projects/ac14f093e9334a2188fde0cc97c57fc4 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:46:58.377063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5b8ff1-8acc-4274-8589-279cb233196a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:46:58.456479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:46:58.470779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5b8ff1-8acc-4274-8589-279cb233196a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:46:58.502398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4054c274-e043-4c66-9303-498d4099b1a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:46:58.503025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4054c274-e043-4c66-9303-498d4099b1a3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:46:58.503305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4054c274-e043-4c66-9303-498d4099b1a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:46:58.504091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4054c274-e043-4c66-9303-498d4099b1a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:46:58.504930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4054c274-e043-4c66-9303-498d4099b1a3 None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=ac14f093e9334a2188fde0cc97c57fc4, user.id=9af76e657f2744aab4685410cbc75d70) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:46:58.505237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4054c274-e043-4c66-9303-498d4099b1a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:46:58.506653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4054c274-e043-4c66-9303-498d4099b1a3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:46:58.512228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 550/1106] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:46:58 2022] GET /identity/v3/role_assignments?user.id=9af76e657f2744aab4685410cbc75d70&scope.project.id=ac14f093e9334a2188fde0cc97c57fc4&role.id=766799d1b6c049168590571f1a959b3b => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:00.652655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b8932-19ea-4fed-ac84-1667079a76ae None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:00.653275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b8932-19ea-4fed-ac84-1667079a76ae None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:00.653275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0b8932-19ea-4fed-ac84-1667079a76ae None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:00.654793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 557/1107] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:00.660032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c1f92c-9325-4cbe-a9f6-bc84addb5c11 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:00.660872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c1f92c-9325-4cbe-a9f6-bc84addb5c11 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:00.661163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c1f92c-9325-4cbe-a9f6-bc84addb5c11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:00.677724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-09c1f92c-9325-4cbe-a9f6-bc84addb5c11 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:00.678916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-09c1f92c-9325-4cbe-a9f6-bc84addb5c11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:00.745266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 551/1108] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:00 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:00.769864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0b8932-19ea-4fed-ac84-1667079a76ae None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:00.868671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0b8932-19ea-4fed-ac84-1667079a76ae None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:00.900496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c53550-8977-43ba-9dee-8e75c806a2f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:00.901297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c53550-8977-43ba-9dee-8e75c806a2f4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:00.901580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c53550-8977-43ba-9dee-8e75c806a2f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:00.901855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c53550-8977-43ba-9dee-8e75c806a2f4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:00.903064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c53550-8977-43ba-9dee-8e75c806a2f4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:00.904452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c53550-8977-43ba-9dee-8e75c806a2f4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:00.906082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 558/1109] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:47:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:00.912326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c1f92c-9325-4cbe-a9f6-bc84addb5c11 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:00.990087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:01.008959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c1f92c-9325-4cbe-a9f6-bc84addb5c11 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:01.041755 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17f96d8-a70a-4d12-ab43-e6033e79b51e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:01.042412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f96d8-a70a-4d12-ab43-e6033e79b51e None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:01.042698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f96d8-a70a-4d12-ab43-e6033e79b51e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:01.042979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f96d8-a70a-4d12-ab43-e6033e79b51e None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:01.043675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f96d8-a70a-4d12-ab43-e6033e79b51e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:01.045143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f96d8-a70a-4d12-ab43-e6033e79b51e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:01.122563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 552/1110] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:47:00 2022] POST /identity/v3/users => generated 286 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:47:03.253306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80be04e7-4ff2-498f-8154-1dfa4229b69c None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:03.253836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80be04e7-4ff2-498f-8154-1dfa4229b69c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:03.253836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80be04e7-4ff2-498f-8154-1dfa4229b69c None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:03.255263 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 559/1111] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:03.260223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642124d1-6f0b-40e2-8758-5582b8cc8f6e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:03.260647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642124d1-6f0b-40e2-8758-5582b8cc8f6e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:03.260869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-642124d1-6f0b-40e2-8758-5582b8cc8f6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:03.278048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-642124d1-6f0b-40e2-8758-5582b8cc8f6e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:03.279235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-642124d1-6f0b-40e2-8758-5582b8cc8f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:03.344563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 553/1112] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:03 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:03.368142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80be04e7-4ff2-498f-8154-1dfa4229b69c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:03.398663 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:03.465452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80be04e7-4ff2-498f-8154-1dfa4229b69c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:03.497615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f89f32-3d39-487b-95e0-5fcae27770cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:03.498448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f89f32-3d39-487b-95e0-5fcae27770cc None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:03.498731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f89f32-3d39-487b-95e0-5fcae27770cc None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:03.499007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f89f32-3d39-487b-95e0-5fcae27770cc None admin] PATH_INFO: `/v3/roles/df56ac8b9c6f46958342cc16f10cea63` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:03.500480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f89f32-3d39-487b-95e0-5fcae27770cc None admin] RBAC: Authorizing `identity:get_role(role_id=df56ac8b9c6f46958342cc16f10cea63)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:03.501904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04f89f32-3d39-487b-95e0-5fcae27770cc None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:03.503277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 560/1113] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:03 2022] GET /identity/v3/roles/df56ac8b9c6f46958342cc16f10cea63 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:03.509165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642124d1-6f0b-40e2-8758-5582b8cc8f6e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:03.572440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:03.605177 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-642124d1-6f0b-40e2-8758-5582b8cc8f6e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:03.637776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7ec624-e299-4bee-b9f7-d79f85af5d90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:03.638637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7ec624-e299-4bee-b9f7-d79f85af5d90 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:03.638921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7ec624-e299-4bee-b9f7-d79f85af5d90 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:03.639198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de7ec624-e299-4bee-b9f7-d79f85af5d90 None admin] PATH_INFO: `/v3/users/2f8731fda412442da8cc5170a98e88d7` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:03.652577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7ec624-e299-4bee-b9f7-d79f85af5d90 None admin] RBAC: Authorizing `identity:get_user(user_id=2f8731fda412442da8cc5170a98e88d7)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:03.654053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de7ec624-e299-4bee-b9f7-d79f85af5d90 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:03.655684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 554/1114] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:03 2022] GET /identity/v3/users/2f8731fda412442da8cc5170a98e88d7 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:03.661437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f89f32-3d39-487b-95e0-5fcae27770cc None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:03.755494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:03.758399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f89f32-3d39-487b-95e0-5fcae27770cc None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:03.791577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afeb77b-2eaf-45a9-abec-4bb40279b0be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:03.792152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afeb77b-2eaf-45a9-abec-4bb40279b0be None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:03.792434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afeb77b-2eaf-45a9-abec-4bb40279b0be None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:03.792709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afeb77b-2eaf-45a9-abec-4bb40279b0be None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:03.859615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afeb77b-2eaf-45a9-abec-4bb40279b0be None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:03.861090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afeb77b-2eaf-45a9-abec-4bb40279b0be None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:03.950710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:03.954075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 561/1115] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:47:03 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:47:03.960561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7ec624-e299-4bee-b9f7-d79f85af5d90 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:04.057667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de7ec624-e299-4bee-b9f7-d79f85af5d90 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:04.079698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:04.089811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7565755c-b8fe-4d83-8285-ca1b3da6d04c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:04.091364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7565755c-b8fe-4d83-8285-ca1b3da6d04c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:04.091364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7565755c-b8fe-4d83-8285-ca1b3da6d04c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:04.091364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7565755c-b8fe-4d83-8285-ca1b3da6d04c None admin] PATH_INFO: `/v3/projects/ac14f093e9334a2188fde0cc97c57fc4` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:04.092331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7565755c-b8fe-4d83-8285-ca1b3da6d04c None admin] RBAC: Authorizing `identity:get_project(project_id=ac14f093e9334a2188fde0cc97c57fc4)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:04.093794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7565755c-b8fe-4d83-8285-ca1b3da6d04c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:04.095459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 555/1116] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:47:03 2022] GET /identity/v3/projects/ac14f093e9334a2188fde0cc97c57fc4 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:04.101585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afeb77b-2eaf-45a9-abec-4bb40279b0be None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:04.196504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afeb77b-2eaf-45a9-abec-4bb40279b0be None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:04.228246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fb494f-59a5-4c43-be00-2af73fec1b3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:04.228908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fb494f-59a5-4c43-be00-2af73fec1b3a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:04.229197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fb494f-59a5-4c43-be00-2af73fec1b3a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:04.229480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fb494f-59a5-4c43-be00-2af73fec1b3a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:04.230299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fb494f-59a5-4c43-be00-2af73fec1b3a None admin] RBAC: Adding query filter params (role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=ac14f093e9334a2188fde0cc97c57fc4, user.id=2f8731fda412442da8cc5170a98e88d7) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:04.230611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fb494f-59a5-4c43-be00-2af73fec1b3a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:04.232026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fb494f-59a5-4c43-be00-2af73fec1b3a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:04.237352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 562/1117] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:47:04 2022] GET /identity/v3/role_assignments?user.id=2f8731fda412442da8cc5170a98e88d7&scope.project.id=ac14f093e9334a2188fde0cc97c57fc4&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 259 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:06.312239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e37f0f7-e262-440f-ab7b-0dbae0143e5c None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:06.312804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e37f0f7-e262-440f-ab7b-0dbae0143e5c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:06.312804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e37f0f7-e262-440f-ab7b-0dbae0143e5c None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:06.314186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 556/1118] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:06.319435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ae9a97-0e51-446f-b043-621d2a2ed582 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:06.319803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ae9a97-0e51-446f-b043-621d2a2ed582 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:06.319997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ae9a97-0e51-446f-b043-621d2a2ed582 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:06.336757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-91ae9a97-0e51-446f-b043-621d2a2ed582 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:06.337952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-91ae9a97-0e51-446f-b043-621d2a2ed582 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:06.392724 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:06.401567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 563/1119] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:06.424387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e37f0f7-e262-440f-ab7b-0dbae0143e5c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:06.521264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e37f0f7-e262-440f-ab7b-0dbae0143e5c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:06.553043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4135ca61-f9a0-4b0f-84ab-51a3e8199e4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:06.553873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4135ca61-f9a0-4b0f-84ab-51a3e8199e4a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:06.554155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4135ca61-f9a0-4b0f-84ab-51a3e8199e4a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:06.554430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4135ca61-f9a0-4b0f-84ab-51a3e8199e4a None admin] PATH_INFO: `/v3/roles/df56ac8b9c6f46958342cc16f10cea63` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:06.555873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4135ca61-f9a0-4b0f-84ab-51a3e8199e4a None admin] RBAC: Authorizing `identity:get_role(role_id=df56ac8b9c6f46958342cc16f10cea63)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:06.557266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4135ca61-f9a0-4b0f-84ab-51a3e8199e4a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:06.558586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 557/1120] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:06 2022] GET /identity/v3/roles/df56ac8b9c6f46958342cc16f10cea63 => generated 223 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:06.564146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ae9a97-0e51-446f-b043-621d2a2ed582 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:06.657870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ae9a97-0e51-446f-b043-621d2a2ed582 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:06.688990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c6be75-8069-47a7-9156-ba727976ef52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:06.689834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c6be75-8069-47a7-9156-ba727976ef52 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:06.690121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c6be75-8069-47a7-9156-ba727976ef52 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:06.690909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c6be75-8069-47a7-9156-ba727976ef52 None admin] PATH_INFO: `/v3/users/2f8731fda412442da8cc5170a98e88d7` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:06.692071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c6be75-8069-47a7-9156-ba727976ef52 None admin] RBAC: Authorizing `identity:get_user(user_id=2f8731fda412442da8cc5170a98e88d7)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:06.693512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c6be75-8069-47a7-9156-ba727976ef52 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:06.695092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 564/1121] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:06 2022] GET /identity/v3/users/2f8731fda412442da8cc5170a98e88d7 => generated 286 bytes in 131 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:06.700725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4135ca61-f9a0-4b0f-84ab-51a3e8199e4a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:06.757261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:06.795659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4135ca61-f9a0-4b0f-84ab-51a3e8199e4a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:06.827940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8629a5a7-4f2d-4242-981a-74c6754b9d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:06.828722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8629a5a7-4f2d-4242-981a-74c6754b9d4f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:06.829036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8629a5a7-4f2d-4242-981a-74c6754b9d4f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:06.829310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8629a5a7-4f2d-4242-981a-74c6754b9d4f None admin] PATH_INFO: `/v3/projects/ac14f093e9334a2188fde0cc97c57fc4` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:06.830466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8629a5a7-4f2d-4242-981a-74c6754b9d4f None admin] RBAC: Authorizing `identity:get_project(project_id=ac14f093e9334a2188fde0cc97c57fc4)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:06.831936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8629a5a7-4f2d-4242-981a-74c6754b9d4f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:06.833631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 558/1122] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:47:06 2022] GET /identity/v3/projects/ac14f093e9334a2188fde0cc97c57fc4 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:06.839443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c6be75-8069-47a7-9156-ba727976ef52 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:06.900664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:06.939009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c6be75-8069-47a7-9156-ba727976ef52 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:06.970608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d93ebf8-6e15-475d-9077-0e66c3b236b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:06.972313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d93ebf8-6e15-475d-9077-0e66c3b236b9 None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:06.972313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d93ebf8-6e15-475d-9077-0e66c3b236b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:06.972599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d93ebf8-6e15-475d-9077-0e66c3b236b9 None admin] PATH_INFO: `/v3/projects/ac14f093e9334a2188fde0cc97c57fc4/users/2f8731fda412442da8cc5170a98e88d7/roles/df56ac8b9c6f46958342cc16f10cea63` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:06.974766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d93ebf8-6e15-475d-9077-0e66c3b236b9 None admin] RBAC: Authorizing `identity:create_grant(project_id=ac14f093e9334a2188fde0cc97c57fc4, user_id=2f8731fda412442da8cc5170a98e88d7, role_id=df56ac8b9c6f46958342cc16f10cea63)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:06.976233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d93ebf8-6e15-475d-9077-0e66c3b236b9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:07.064236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 565/1123] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:47:06 2022] PUT /identity/v3/projects/ac14f093e9334a2188fde0cc97c57fc4/users/2f8731fda412442da8cc5170a98e88d7/roles/df56ac8b9c6f46958342cc16f10cea63 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 12:47:09.131544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23de3677-f155-4f27-8bdc-d80aefd2b737 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:09.132079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23de3677-f155-4f27-8bdc-d80aefd2b737 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:09.132079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23de3677-f155-4f27-8bdc-d80aefd2b737 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:09.133492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 559/1124] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:09.138582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37acba70-b267-4ee5-90f2-f6714dba8a97 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:09.138989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37acba70-b267-4ee5-90f2-f6714dba8a97 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:09.139143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37acba70-b267-4ee5-90f2-f6714dba8a97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:09.156145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-37acba70-b267-4ee5-90f2-f6714dba8a97 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:09.157339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-37acba70-b267-4ee5-90f2-f6714dba8a97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:09.222614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 566/1125] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:09 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:47:09.246676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23de3677-f155-4f27-8bdc-d80aefd2b737 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:09.345535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23de3677-f155-4f27-8bdc-d80aefd2b737 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:09.370675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:09.378679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a9a8c9-e192-47ee-8c59-a0fd764faa40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:09.379549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a9a8c9-e192-47ee-8c59-a0fd764faa40 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:09.379855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a9a8c9-e192-47ee-8c59-a0fd764faa40 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:09.380130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a9a8c9-e192-47ee-8c59-a0fd764faa40 None admin] PATH_INFO: `/v3/roles/df56ac8b9c6f46958342cc16f10cea63` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:09.381654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a9a8c9-e192-47ee-8c59-a0fd764faa40 None admin] RBAC: Authorizing `identity:get_role(role_id=df56ac8b9c6f46958342cc16f10cea63)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:09.383027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a9a8c9-e192-47ee-8c59-a0fd764faa40 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:09.384386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 560/1126] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:09 2022] GET /identity/v3/roles/df56ac8b9c6f46958342cc16f10cea63 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:09.390022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37acba70-b267-4ee5-90f2-f6714dba8a97 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:09.439148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:09.490822 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37acba70-b267-4ee5-90f2-f6714dba8a97 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:09.524391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94848f9f-d6e5-4bd4-bf00-48c47fb51912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:09.525244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94848f9f-d6e5-4bd4-bf00-48c47fb51912 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:09.525526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94848f9f-d6e5-4bd4-bf00-48c47fb51912 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:09.525806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94848f9f-d6e5-4bd4-bf00-48c47fb51912 None admin] PATH_INFO: `/v3/users/2f8731fda412442da8cc5170a98e88d7` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:09.527000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94848f9f-d6e5-4bd4-bf00-48c47fb51912 None admin] RBAC: Authorizing `identity:get_user(user_id=2f8731fda412442da8cc5170a98e88d7)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:09.528438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94848f9f-d6e5-4bd4-bf00-48c47fb51912 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:09.530092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 567/1127] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:09 2022] GET /identity/v3/users/2f8731fda412442da8cc5170a98e88d7 => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:09.535817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a9a8c9-e192-47ee-8c59-a0fd764faa40 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:09.632593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a9a8c9-e192-47ee-8c59-a0fd764faa40 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:09.664833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9decef3-50f1-4751-967a-d922df9eec55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:09.665469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9decef3-50f1-4751-967a-d922df9eec55 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:09.665758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9decef3-50f1-4751-967a-d922df9eec55 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:09.666039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9decef3-50f1-4751-967a-d922df9eec55 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:09.724204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:09.730953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9decef3-50f1-4751-967a-d922df9eec55 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:09.736630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9decef3-50f1-4751-967a-d922df9eec55 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:09.832373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 561/1128] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:47:09 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:47:09.838220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94848f9f-d6e5-4bd4-bf00-48c47fb51912 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:09.937347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94848f9f-d6e5-4bd4-bf00-48c47fb51912 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:09.956877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:09.970058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b2231-3b70-4df0-a5ec-73c569feddd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:09.970901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b2231-3b70-4df0-a5ec-73c569feddd4 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:09.971184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b2231-3b70-4df0-a5ec-73c569feddd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:09.971503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b2231-3b70-4df0-a5ec-73c569feddd4 None admin] PATH_INFO: `/v3/projects/ac14f093e9334a2188fde0cc97c57fc4` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:09.972701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b2231-3b70-4df0-a5ec-73c569feddd4 None admin] RBAC: Authorizing `identity:get_project(project_id=ac14f093e9334a2188fde0cc97c57fc4)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:09.974167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b2231-3b70-4df0-a5ec-73c569feddd4 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:09.975888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 568/1129] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:47:09 2022] GET /identity/v3/projects/ac14f093e9334a2188fde0cc97c57fc4 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:09.981921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9decef3-50f1-4751-967a-d922df9eec55 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:10.072323 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:10.077767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9decef3-50f1-4751-967a-d922df9eec55 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:10.118797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0176ad4-04d0-4f83-b190-ea45a84041eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:10.119445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0176ad4-04d0-4f83-b190-ea45a84041eb None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:10.119725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0176ad4-04d0-4f83-b190-ea45a84041eb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:10.119995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0176ad4-04d0-4f83-b190-ea45a84041eb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:10.120832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0176ad4-04d0-4f83-b190-ea45a84041eb None admin] RBAC: Adding query filter params (role.id=df56ac8b9c6f46958342cc16f10cea63, scope.project.id=ac14f093e9334a2188fde0cc97c57fc4, user.id=2f8731fda412442da8cc5170a98e88d7) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:10.121145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0176ad4-04d0-4f83-b190-ea45a84041eb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:10.122567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0176ad4-04d0-4f83-b190-ea45a84041eb None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:10.128908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 562/1130] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:47:09 2022] GET /identity/v3/role_assignments?user.id=2f8731fda412442da8cc5170a98e88d7&scope.project.id=ac14f093e9334a2188fde0cc97c57fc4&role.id=df56ac8b9c6f46958342cc16f10cea63 => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:12.268977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bfe14-a6cb-4f22-bef1-36de020d6ecc None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:12.269526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bfe14-a6cb-4f22-bef1-36de020d6ecc None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:12.269526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-917bfe14-a6cb-4f22-bef1-36de020d6ecc None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:12.270887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 569/1131] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:12.276274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da5d0a4-6bfe-4f44-af51-351a242d344d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:12.276705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da5d0a4-6bfe-4f44-af51-351a242d344d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:12.276842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da5d0a4-6bfe-4f44-af51-351a242d344d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:12.293760 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1da5d0a4-6bfe-4f44-af51-351a242d344d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:12.294925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1da5d0a4-6bfe-4f44-af51-351a242d344d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:12.359760 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 563/1132] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:12 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:12.382709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917bfe14-a6cb-4f22-bef1-36de020d6ecc None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:12.479301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-917bfe14-a6cb-4f22-bef1-36de020d6ecc None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:12.510778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43d870a-95fa-449b-804b-d4a3fd2760d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:12.511537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43d870a-95fa-449b-804b-d4a3fd2760d7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:12.511817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43d870a-95fa-449b-804b-d4a3fd2760d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:12.512089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43d870a-95fa-449b-804b-d4a3fd2760d7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:12.513262 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43d870a-95fa-449b-804b-d4a3fd2760d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:12.514680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43d870a-95fa-449b-804b-d4a3fd2760d7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:12.516245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 570/1133] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:47:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:12.522411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da5d0a4-6bfe-4f44-af51-351a242d344d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:12.607247 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:12.617197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da5d0a4-6bfe-4f44-af51-351a242d344d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:12.652151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9cd86db-50ac-44bc-8c05-b997b4f80683 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:12.652829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cd86db-50ac-44bc-8c05-b997b4f80683 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:12.653201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cd86db-50ac-44bc-8c05-b997b4f80683 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:12.653506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9cd86db-50ac-44bc-8c05-b997b4f80683 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:12.654279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cd86db-50ac-44bc-8c05-b997b4f80683 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:12.655702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9cd86db-50ac-44bc-8c05-b997b4f80683 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:12.757895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 564/1134] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:47:12 2022] POST /identity/v3/projects => generated 311 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:47:14.867810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff796319-8b94-423a-b697-30aa4481fc4a None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:14.868357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff796319-8b94-423a-b697-30aa4481fc4a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:14.868357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff796319-8b94-423a-b697-30aa4481fc4a None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:14.869770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 571/1135] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:14.875016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46520cff-8dd2-429a-9ae2-cbe47fd3e480 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:14.875440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46520cff-8dd2-429a-9ae2-cbe47fd3e480 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:14.875579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46520cff-8dd2-429a-9ae2-cbe47fd3e480 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:14.892388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-46520cff-8dd2-429a-9ae2-cbe47fd3e480 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:14.894138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-46520cff-8dd2-429a-9ae2-cbe47fd3e480 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:14.958178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 565/1136] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:14 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:14.982180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff796319-8b94-423a-b697-30aa4481fc4a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:15.037114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:15.078949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff796319-8b94-423a-b697-30aa4481fc4a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:15.110700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1180d10f-8100-41cf-ab16-693a6b746a0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:15.111469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1180d10f-8100-41cf-ab16-693a6b746a0a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:15.111776 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1180d10f-8100-41cf-ab16-693a6b746a0a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:15.112055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1180d10f-8100-41cf-ab16-693a6b746a0a None admin] PATH_INFO: `/v3/domains/default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:15.113236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1180d10f-8100-41cf-ab16-693a6b746a0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:15.114668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1180d10f-8100-41cf-ab16-693a6b746a0a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:15.116244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 572/1137] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:47:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:15.121952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46520cff-8dd2-429a-9ae2-cbe47fd3e480 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:15.193542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:15.221975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46520cff-8dd2-429a-9ae2-cbe47fd3e480 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:15.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-469a737d-9a5c-4ff1-9b80-7c34d65115c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:15.255783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a737d-9a5c-4ff1-9b80-7c34d65115c8 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:15.256071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a737d-9a5c-4ff1-9b80-7c34d65115c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:15.256355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-469a737d-9a5c-4ff1-9b80-7c34d65115c8 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:15.257100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469a737d-9a5c-4ff1-9b80-7c34d65115c8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:15.258533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-469a737d-9a5c-4ff1-9b80-7c34d65115c8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:15.362637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 566/1138] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:47:15 2022] POST /identity/v3/users => generated 286 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:47:17.540479 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd587c55-db41-441c-bcaa-7e5b5e396ace None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:17.541034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd587c55-db41-441c-bcaa-7e5b5e396ace None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:17.541119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd587c55-db41-441c-bcaa-7e5b5e396ace None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:17.542423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 573/1139] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:17.547705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab51707-1ccd-421f-94bd-e6677c1affe3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:17.548083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab51707-1ccd-421f-94bd-e6677c1affe3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:17.548268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab51707-1ccd-421f-94bd-e6677c1affe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:17.567106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5ab51707-1ccd-421f-94bd-e6677c1affe3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:17.568297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5ab51707-1ccd-421f-94bd-e6677c1affe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:17.635988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 567/1140] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:17 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:17.664386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd587c55-db41-441c-bcaa-7e5b5e396ace None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:17.782724 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd587c55-db41-441c-bcaa-7e5b5e396ace None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:17.809996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:17.824419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cda33a-c30e-4b77-9d5c-8bbaf8530b0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:17.825222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cda33a-c30e-4b77-9d5c-8bbaf8530b0f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:17.825887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cda33a-c30e-4b77-9d5c-8bbaf8530b0f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:17.826161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2cda33a-c30e-4b77-9d5c-8bbaf8530b0f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:17.836051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2cda33a-c30e-4b77-9d5c-8bbaf8530b0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-e2cda33a-c30e-4b77-9d5c-8bbaf8530b0f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:47:17.848179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:47:17.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:47:17.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 574/1141] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:47:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:47:17.855738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab51707-1ccd-421f-94bd-e6677c1affe3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:17.916649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:17.961374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab51707-1ccd-421f-94bd-e6677c1affe3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:17.999239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e7f5c0-4615-4a31-b34e-5a22d328f99e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:17.999905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7f5c0-4615-4a31-b34e-5a22d328f99e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:18.000189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7f5c0-4615-4a31-b34e-5a22d328f99e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:18.000468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7f5c0-4615-4a31-b34e-5a22d328f99e None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:18.001242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e7f5c0-4615-4a31-b34e-5a22d328f99e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:18.001550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e7f5c0-4615-4a31-b34e-5a22d328f99e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:18.002922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e7f5c0-4615-4a31-b34e-5a22d328f99e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:18.010701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 568/1142] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:47:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:18.016425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cda33a-c30e-4b77-9d5c-8bbaf8530b0f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:18.149339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2cda33a-c30e-4b77-9d5c-8bbaf8530b0f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:18.181773 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6d6af1-4790-4849-90be-ee5f69a7cc36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:18.182553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6d6af1-4790-4849-90be-ee5f69a7cc36 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:18.182836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6d6af1-4790-4849-90be-ee5f69a7cc36 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:18.183111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6d6af1-4790-4849-90be-ee5f69a7cc36 None admin] PATH_INFO: `/v3/users/74059a2284d543e6b1be9fa4914f2a79` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:18.196143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6d6af1-4790-4849-90be-ee5f69a7cc36 None admin] RBAC: Authorizing `identity:get_user(user_id=74059a2284d543e6b1be9fa4914f2a79)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:18.197633 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6d6af1-4790-4849-90be-ee5f69a7cc36 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:18.199293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 575/1143] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:18 2022] GET /identity/v3/users/74059a2284d543e6b1be9fa4914f2a79 => generated 286 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:18.205083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e7f5c0-4615-4a31-b34e-5a22d328f99e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:18.301212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e7f5c0-4615-4a31-b34e-5a22d328f99e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:18.319853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:18.333965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddc39da-da1f-4956-8764-36572c06845f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:18.334608 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddc39da-da1f-4956-8764-36572c06845f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:18.334891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddc39da-da1f-4956-8764-36572c06845f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:18.335166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddc39da-da1f-4956-8764-36572c06845f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:18.399809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddc39da-da1f-4956-8764-36572c06845f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:18.401293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddc39da-da1f-4956-8764-36572c06845f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:18.495504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 569/1144] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:47:18 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:47:18.501441 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6d6af1-4790-4849-90be-ee5f69a7cc36 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:18.543260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:18.596643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6d6af1-4790-4849-90be-ee5f69a7cc36 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:18.628904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4073e36-1ec6-4611-9a0f-39dd3fc6c13d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:18.629744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4073e36-1ec6-4611-9a0f-39dd3fc6c13d None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:18.630024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4073e36-1ec6-4611-9a0f-39dd3fc6c13d None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:18.630299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4073e36-1ec6-4611-9a0f-39dd3fc6c13d None admin] PATH_INFO: `/v3/projects/55031496f4ae4d0ab196644299830780` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:18.631443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4073e36-1ec6-4611-9a0f-39dd3fc6c13d None admin] RBAC: Authorizing `identity:get_project(project_id=55031496f4ae4d0ab196644299830780)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:18.632891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4073e36-1ec6-4611-9a0f-39dd3fc6c13d None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:18.634544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 576/1145] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:47:18 2022] GET /identity/v3/projects/55031496f4ae4d0ab196644299830780 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:18.640639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddc39da-da1f-4956-8764-36572c06845f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:18.661722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:18.735602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddc39da-da1f-4956-8764-36572c06845f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:18.768404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c67b903-09d3-44ef-9a97-de6a8373472f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:18.769004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c67b903-09d3-44ef-9a97-de6a8373472f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:18.769289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c67b903-09d3-44ef-9a97-de6a8373472f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:18.769568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c67b903-09d3-44ef-9a97-de6a8373472f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:18.770387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c67b903-09d3-44ef-9a97-de6a8373472f None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=55031496f4ae4d0ab196644299830780, user.id=74059a2284d543e6b1be9fa4914f2a79) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:18.770701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c67b903-09d3-44ef-9a97-de6a8373472f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:18.772125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c67b903-09d3-44ef-9a97-de6a8373472f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:18.777630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 570/1146] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:47:18 2022] GET /identity/v3/role_assignments?user.id=74059a2284d543e6b1be9fa4914f2a79&scope.project.id=55031496f4ae4d0ab196644299830780&role.id=766799d1b6c049168590571f1a959b3b => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:20.827916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf3a3e-9edb-4686-a9b9-903e196146bf None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:20.828440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf3a3e-9edb-4686-a9b9-903e196146bf None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:20.828440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bf3a3e-9edb-4686-a9b9-903e196146bf None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:20.829908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 577/1147] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:20.835167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6370ca-72a7-4031-8d8b-b65098926405 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:20.835585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6370ca-72a7-4031-8d8b-b65098926405 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:20.835732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6370ca-72a7-4031-8d8b-b65098926405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:20.852399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-aa6370ca-72a7-4031-8d8b-b65098926405 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:20.853634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-aa6370ca-72a7-4031-8d8b-b65098926405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:20.923571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 571/1148] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:20 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:20.946215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf3a3e-9edb-4686-a9b9-903e196146bf None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:21.047820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bf3a3e-9edb-4686-a9b9-903e196146bf None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:21.058421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:21.080035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe13762-7ff9-44a6-a984-cd1b74d749c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:21.080753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe13762-7ff9-44a6-a984-cd1b74d749c9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:21.081088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe13762-7ff9-44a6-a984-cd1b74d749c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:21.081360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe13762-7ff9-44a6-a984-cd1b74d749c9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:21.090841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe13762-7ff9-44a6-a984-cd1b74d749c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-3fe13762-7ff9-44a6-a984-cd1b74d749c9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:47:21.099533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:47:21.100606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:47:21.101522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 578/1149] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:47:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:47:21.107061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6370ca-72a7-4031-8d8b-b65098926405 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:21.122434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:21.202027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6370ca-72a7-4031-8d8b-b65098926405 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:21.235265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6659ef25-2a60-47b8-9ec7-5d4feee37286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:21.235924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6659ef25-2a60-47b8-9ec7-5d4feee37286 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:21.236227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6659ef25-2a60-47b8-9ec7-5d4feee37286 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:21.236502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6659ef25-2a60-47b8-9ec7-5d4feee37286 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:21.237266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6659ef25-2a60-47b8-9ec7-5d4feee37286 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:21.237568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6659ef25-2a60-47b8-9ec7-5d4feee37286 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:21.238939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6659ef25-2a60-47b8-9ec7-5d4feee37286 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:21.246809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 572/1150] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:47:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:21.252566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe13762-7ff9-44a6-a984-cd1b74d749c9 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:21.348161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe13762-7ff9-44a6-a984-cd1b74d749c9 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:21.379388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdb2db-bf96-4be9-8815-1e18fa731338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:21.380231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdb2db-bf96-4be9-8815-1e18fa731338 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:21.380510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdb2db-bf96-4be9-8815-1e18fa731338 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:21.380800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdb2db-bf96-4be9-8815-1e18fa731338 None admin] PATH_INFO: `/v3/users/74059a2284d543e6b1be9fa4914f2a79` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:21.382009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bdb2db-bf96-4be9-8815-1e18fa731338 None admin] RBAC: Authorizing `identity:get_user(user_id=74059a2284d543e6b1be9fa4914f2a79)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:21.383429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bdb2db-bf96-4be9-8815-1e18fa731338 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:21.385066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 579/1151] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:21 2022] GET /identity/v3/users/74059a2284d543e6b1be9fa4914f2a79 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:21.390717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6659ef25-2a60-47b8-9ec7-5d4feee37286 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:21.464486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:21.486613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6659ef25-2a60-47b8-9ec7-5d4feee37286 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:21.518361 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7274b8ad-88d7-4357-b4c0-ad1b95977b62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:21.519194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7274b8ad-88d7-4357-b4c0-ad1b95977b62 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:21.519474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7274b8ad-88d7-4357-b4c0-ad1b95977b62 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:21.519749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7274b8ad-88d7-4357-b4c0-ad1b95977b62 None admin] PATH_INFO: `/v3/projects/55031496f4ae4d0ab196644299830780` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:21.520930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7274b8ad-88d7-4357-b4c0-ad1b95977b62 None admin] RBAC: Authorizing `identity:get_project(project_id=55031496f4ae4d0ab196644299830780)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:21.522779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7274b8ad-88d7-4357-b4c0-ad1b95977b62 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:21.524433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 573/1152] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:47:21 2022] GET /identity/v3/projects/55031496f4ae4d0ab196644299830780 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:47:21.530165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdb2db-bf96-4be9-8815-1e18fa731338 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:21.568328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:21.625166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdb2db-bf96-4be9-8815-1e18fa731338 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:21.658209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f396a140-7b5e-4478-b3c8-c154aac812d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:21.659068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f396a140-7b5e-4478-b3c8-c154aac812d1 None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:21.659345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f396a140-7b5e-4478-b3c8-c154aac812d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:21.659612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f396a140-7b5e-4478-b3c8-c154aac812d1 None admin] PATH_INFO: `/v3/projects/55031496f4ae4d0ab196644299830780/users/74059a2284d543e6b1be9fa4914f2a79/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:21.661532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f396a140-7b5e-4478-b3c8-c154aac812d1 None admin] RBAC: Authorizing `identity:create_grant(project_id=55031496f4ae4d0ab196644299830780, user_id=74059a2284d543e6b1be9fa4914f2a79, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:21.662937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f396a140-7b5e-4478-b3c8-c154aac812d1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:21.733030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 580/1153] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:47:21 2022] PUT /identity/v3/projects/55031496f4ae4d0ab196644299830780/users/74059a2284d543e6b1be9fa4914f2a79/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:47:23.831073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a808ae3-35c2-4812-a3b9-a8d4f7bbf3df None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:23.831598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a808ae3-35c2-4812-a3b9-a8d4f7bbf3df None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:23.831691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a808ae3-35c2-4812-a3b9-a8d4f7bbf3df None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:23.833043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 574/1154] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:23.838978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd286fab-3be0-474f-b8e4-34c9d1c96f5c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:23.839487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd286fab-3be0-474f-b8e4-34c9d1c96f5c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:23.839670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd286fab-3be0-474f-b8e4-34c9d1c96f5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:23.856193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-cd286fab-3be0-474f-b8e4-34c9d1c96f5c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:23.857380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-cd286fab-3be0-474f-b8e4-34c9d1c96f5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:23.919786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 581/1155] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:23 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:47:23.942756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a808ae3-35c2-4812-a3b9-a8d4f7bbf3df None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:24.043596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a808ae3-35c2-4812-a3b9-a8d4f7bbf3df None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:24.075850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b58f1a-5ba5-45fd-9602-25071fa402d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:24.076620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b58f1a-5ba5-45fd-9602-25071fa402d5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:24.076927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b58f1a-5ba5-45fd-9602-25071fa402d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:24.077200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b58f1a-5ba5-45fd-9602-25071fa402d5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:24.087304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34b58f1a-5ba5-45fd-9602-25071fa402d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-34b58f1a-5ba5-45fd-9602-25071fa402d5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:47:24.094671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:47:24.096261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:47:24.097191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 575/1156] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:47:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:47:24.102385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd286fab-3be0-474f-b8e4-34c9d1c96f5c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:24.129241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:24.198065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd286fab-3be0-474f-b8e4-34c9d1c96f5c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:24.230753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c21d3-c660-48a5-96a6-cf7a9409cfb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:24.231403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c21d3-c660-48a5-96a6-cf7a9409cfb9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:24.231689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c21d3-c660-48a5-96a6-cf7a9409cfb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:24.231970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0c21d3-c660-48a5-96a6-cf7a9409cfb9 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:24.232709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0c21d3-c660-48a5-96a6-cf7a9409cfb9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:24.233046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0c21d3-c660-48a5-96a6-cf7a9409cfb9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:24.234412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0c21d3-c660-48a5-96a6-cf7a9409cfb9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:24.242196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 582/1157] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:47:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:24.247836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b58f1a-5ba5-45fd-9602-25071fa402d5 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:24.251515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:24.343989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34b58f1a-5ba5-45fd-9602-25071fa402d5 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:24.376329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5338c2e2-bede-4d6d-a42b-abec4fc46f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:24.377239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5338c2e2-bede-4d6d-a42b-abec4fc46f3f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:24.377521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5338c2e2-bede-4d6d-a42b-abec4fc46f3f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:24.377795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5338c2e2-bede-4d6d-a42b-abec4fc46f3f None admin] PATH_INFO: `/v3/users/74059a2284d543e6b1be9fa4914f2a79` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:24.378959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5338c2e2-bede-4d6d-a42b-abec4fc46f3f None admin] RBAC: Authorizing `identity:get_user(user_id=74059a2284d543e6b1be9fa4914f2a79)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:24.380386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5338c2e2-bede-4d6d-a42b-abec4fc46f3f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:24.382054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 576/1158] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:24 2022] GET /identity/v3/users/74059a2284d543e6b1be9fa4914f2a79 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:24.387704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c21d3-c660-48a5-96a6-cf7a9409cfb9 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:24.481553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:24.490612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0c21d3-c660-48a5-96a6-cf7a9409cfb9 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:24.523558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0043585c-c3e3-44d9-b4e5-f97d826b7e16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:24.524197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0043585c-c3e3-44d9-b4e5-f97d826b7e16 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:24.524632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0043585c-c3e3-44d9-b4e5-f97d826b7e16 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:24.524949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0043585c-c3e3-44d9-b4e5-f97d826b7e16 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:24.588452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0043585c-c3e3-44d9-b4e5-f97d826b7e16 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:24.589936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0043585c-c3e3-44d9-b4e5-f97d826b7e16 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:24.674160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:24.683826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 583/1159] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:47:24 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:47:24.690402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5338c2e2-bede-4d6d-a42b-abec4fc46f3f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:24.757946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:24.794675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5338c2e2-bede-4d6d-a42b-abec4fc46f3f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:24.828153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2d4e783-c317-4675-ab32-6be67f9f7c5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:24.829033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d4e783-c317-4675-ab32-6be67f9f7c5c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:24.829317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d4e783-c317-4675-ab32-6be67f9f7c5c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:24.829593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2d4e783-c317-4675-ab32-6be67f9f7c5c None admin] PATH_INFO: `/v3/projects/55031496f4ae4d0ab196644299830780` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:24.831355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d4e783-c317-4675-ab32-6be67f9f7c5c None admin] RBAC: Authorizing `identity:get_project(project_id=55031496f4ae4d0ab196644299830780)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:24.832823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2d4e783-c317-4675-ab32-6be67f9f7c5c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:24.834522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 577/1160] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:47:24 2022] GET /identity/v3/projects/55031496f4ae4d0ab196644299830780 => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:24.840579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0043585c-c3e3-44d9-b4e5-f97d826b7e16 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:24.933779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0043585c-c3e3-44d9-b4e5-f97d826b7e16 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:24.966563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87802ce7-6a91-4f47-86b5-3ecf687f658c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:24.967193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87802ce7-6a91-4f47-86b5-3ecf687f658c None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:24.967472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87802ce7-6a91-4f47-86b5-3ecf687f658c None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:24.967744 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87802ce7-6a91-4f47-86b5-3ecf687f658c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:24.968551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87802ce7-6a91-4f47-86b5-3ecf687f658c None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=55031496f4ae4d0ab196644299830780, user.id=74059a2284d543e6b1be9fa4914f2a79) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:24.968886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87802ce7-6a91-4f47-86b5-3ecf687f658c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:24.970320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87802ce7-6a91-4f47-86b5-3ecf687f658c None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:24.975829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 584/1161] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:47:24 2022] GET /identity/v3/role_assignments?user.id=74059a2284d543e6b1be9fa4914f2a79&scope.project.id=55031496f4ae4d0ab196644299830780&role.id=766799d1b6c049168590571f1a959b3b => generated 608 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:27.126665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cfdafa-20ba-404c-8939-86e05ffdeb93 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:27.127197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cfdafa-20ba-404c-8939-86e05ffdeb93 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:27.127306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2cfdafa-20ba-404c-8939-86e05ffdeb93 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:27.128595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 578/1162] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:27.133801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404818cc-ce80-467a-93b0-cc3d038700ab None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:27.134217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404818cc-ce80-467a-93b0-cc3d038700ab None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:27.134356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404818cc-ce80-467a-93b0-cc3d038700ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:27.150620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-404818cc-ce80-467a-93b0-cc3d038700ab None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:27.151806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-404818cc-ce80-467a-93b0-cc3d038700ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:27.198490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:27.215281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 585/1163] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:27 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:47:27.237832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cfdafa-20ba-404c-8939-86e05ffdeb93 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:27.337776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2cfdafa-20ba-404c-8939-86e05ffdeb93 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:27.370165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e8650-7ff0-4aaf-b84c-32f405291c16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:27.370794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e8650-7ff0-4aaf-b84c-32f405291c16 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:27.371076 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e8650-7ff0-4aaf-b84c-32f405291c16 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:27.371353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68e8650-7ff0-4aaf-b84c-32f405291c16 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:27.382696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:27.436422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e8650-7ff0-4aaf-b84c-32f405291c16 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:27.437881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68e8650-7ff0-4aaf-b84c-32f405291c16 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:27.530792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 579/1164] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:47:27 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:47:27.536723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404818cc-ce80-467a-93b0-cc3d038700ab None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:27.629910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404818cc-ce80-467a-93b0-cc3d038700ab None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:27.662402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe444805-0718-4c8a-b794-1973845386a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:27.663122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe444805-0718-4c8a-b794-1973845386a3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:27.663401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe444805-0718-4c8a-b794-1973845386a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:27.663671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe444805-0718-4c8a-b794-1973845386a3 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:27.669975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe444805-0718-4c8a-b794-1973845386a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:27.671400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe444805-0718-4c8a-b794-1973845386a3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-fe444805-0718-4c8a-b794-1973845386a3 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:47:27.685604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:47:27.691244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:47:27.692359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 586/1165] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:47:27 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:47:27.693422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e8650-7ff0-4aaf-b84c-32f405291c16 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:27.731711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:27.787357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68e8650-7ff0-4aaf-b84c-32f405291c16 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:27.819503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957851a7-2d20-4391-9027-146cc7678ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:27.820147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957851a7-2d20-4391-9027-146cc7678ded None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:27.820430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957851a7-2d20-4391-9027-146cc7678ded None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:27.820708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957851a7-2d20-4391-9027-146cc7678ded None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:27.821443 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957851a7-2d20-4391-9027-146cc7678ded None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:27.821748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957851a7-2d20-4391-9027-146cc7678ded None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:27.823110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957851a7-2d20-4391-9027-146cc7678ded None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:27.830050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 580/1166] 10.222.0.25 () {56 vars in 1115 bytes} [Fri Jun 10 12:47:27 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:29.912406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e433d5eb-68f6-4814-bb93-2661ab423851 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:29.912990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e433d5eb-68f6-4814-bb93-2661ab423851 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:29.912990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e433d5eb-68f6-4814-bb93-2661ab423851 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:29.914337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 587/1167] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:29.919507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dd59bd-f0f6-4e15-8c00-35b5d097711a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:29.919875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dd59bd-f0f6-4e15-8c00-35b5d097711a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:29.920076 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9dd59bd-f0f6-4e15-8c00-35b5d097711a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:29.936249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a9dd59bd-f0f6-4e15-8c00-35b5d097711a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:29.937474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a9dd59bd-f0f6-4e15-8c00-35b5d097711a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:30.003835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 581/1168] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:29 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:30.027337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e433d5eb-68f6-4814-bb93-2661ab423851 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:30.072581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:30.124588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e433d5eb-68f6-4814-bb93-2661ab423851 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:30.157455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b71232e-a079-4093-a4c8-80c56dfabe15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:30.158122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b71232e-a079-4093-a4c8-80c56dfabe15 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:30.158406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b71232e-a079-4093-a4c8-80c56dfabe15 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:30.158683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b71232e-a079-4093-a4c8-80c56dfabe15 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:30.159421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b71232e-a079-4093-a4c8-80c56dfabe15 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:30.160808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b71232e-a079-4093-a4c8-80c56dfabe15 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:30.287487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 588/1169] 10.222.0.25 () {60 vars in 1138 bytes} [Fri Jun 10 12:47:30 2022] POST /identity/v3/domains => generated 267 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:47:32.440484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c6b6a9-290c-415f-bab7-d77bdfdd1ca5 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:32.441095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c6b6a9-290c-415f-bab7-d77bdfdd1ca5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:32.441095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c6b6a9-290c-415f-bab7-d77bdfdd1ca5 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:32.442450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 582/1170] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:32.447685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42df0461-7b06-4864-93fe-c68f7038b233 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:32.448128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42df0461-7b06-4864-93fe-c68f7038b233 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:32.448230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42df0461-7b06-4864-93fe-c68f7038b233 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:32.465720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-42df0461-7b06-4864-93fe-c68f7038b233 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:32.466885 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-42df0461-7b06-4864-93fe-c68f7038b233 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:32.531578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 589/1171] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:32 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:32.555694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c6b6a9-290c-415f-bab7-d77bdfdd1ca5 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:32.589207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:32.652252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c6b6a9-290c-415f-bab7-d77bdfdd1ca5 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:32.684052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b2dd99-da1f-4d1c-94c8-bb5d930d3616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:32.684916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b2dd99-da1f-4d1c-94c8-bb5d930d3616 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:32.685201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b2dd99-da1f-4d1c-94c8-bb5d930d3616 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:32.685471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b2dd99-da1f-4d1c-94c8-bb5d930d3616 None admin] PATH_INFO: `/v3/domains/a8b72b8cb5014c23905b8d2a5bebe7ed` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:32.695383 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b2dd99-da1f-4d1c-94c8-bb5d930d3616 None admin] RBAC: Authorizing `identity:get_domain(domain_id=a8b72b8cb5014c23905b8d2a5bebe7ed)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:32.696848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b2dd99-da1f-4d1c-94c8-bb5d930d3616 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:32.698443 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 583/1172] 10.222.0.25 () {56 vars in 1183 bytes} [Fri Jun 10 12:47:32 2022] GET /identity/v3/domains/a8b72b8cb5014c23905b8d2a5bebe7ed => generated 267 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:32.704735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42df0461-7b06-4864-93fe-c68f7038b233 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:32.736909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:32.798614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42df0461-7b06-4864-93fe-c68f7038b233 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:32.831446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35413501-896c-4002-a6d7-676bddb680dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:32.832085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35413501-896c-4002-a6d7-676bddb680dc None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:32.832368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35413501-896c-4002-a6d7-676bddb680dc None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:32.832645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35413501-896c-4002-a6d7-676bddb680dc None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:32.833370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35413501-896c-4002-a6d7-676bddb680dc None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:32.834787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35413501-896c-4002-a6d7-676bddb680dc None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:32.927620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 590/1173] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:47:32 2022] POST /identity/v3/projects => generated 361 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:47:35.081227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02418db1-fb55-4d36-b3c8-5002ac62acb2 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:35.081780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02418db1-fb55-4d36-b3c8-5002ac62acb2 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:35.081780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02418db1-fb55-4d36-b3c8-5002ac62acb2 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:35.083179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 584/1174] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:35.088200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c28ecb7-590f-4f41-a6da-63c65f743603 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:35.088616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c28ecb7-590f-4f41-a6da-63c65f743603 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:35.088752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c28ecb7-590f-4f41-a6da-63c65f743603 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:35.105544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6c28ecb7-590f-4f41-a6da-63c65f743603 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:35.106701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6c28ecb7-590f-4f41-a6da-63c65f743603 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:35.169456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 591/1175] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:35 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:35.193763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02418db1-fb55-4d36-b3c8-5002ac62acb2 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:35.284999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:35.292185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02418db1-fb55-4d36-b3c8-5002ac62acb2 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:35.331501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc0bc61-df81-4f60-9552-84f7b393885a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:35.332284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc0bc61-df81-4f60-9552-84f7b393885a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:35.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc0bc61-df81-4f60-9552-84f7b393885a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:35.332883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc0bc61-df81-4f60-9552-84f7b393885a None admin] PATH_INFO: `/v3/domains/a8b72b8cb5014c23905b8d2a5bebe7ed` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:35.334054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc0bc61-df81-4f60-9552-84f7b393885a None admin] RBAC: Authorizing `identity:get_domain(domain_id=a8b72b8cb5014c23905b8d2a5bebe7ed)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:35.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc0bc61-df81-4f60-9552-84f7b393885a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:35.337175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 585/1176] 10.222.0.25 () {56 vars in 1183 bytes} [Fri Jun 10 12:47:35 2022] GET /identity/v3/domains/a8b72b8cb5014c23905b8d2a5bebe7ed => generated 267 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:47:35.342748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c28ecb7-590f-4f41-a6da-63c65f743603 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:35.359817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:35.437640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c28ecb7-590f-4f41-a6da-63c65f743603 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:35.469318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fcd3a69-8bfd-4c3d-b8a2-8716c78e0974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:35.470512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcd3a69-8bfd-4c3d-b8a2-8716c78e0974 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:35.470789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcd3a69-8bfd-4c3d-b8a2-8716c78e0974 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:35.471062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fcd3a69-8bfd-4c3d-b8a2-8716c78e0974 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:35.471750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fcd3a69-8bfd-4c3d-b8a2-8716c78e0974 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:35.473217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fcd3a69-8bfd-4c3d-b8a2-8716c78e0974 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:35.561334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 592/1177] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:47:35 2022] POST /identity/v3/users => generated 311 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:47:37.677613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea7df4e-3a3d-40d2-9d30-4cb054c3770d None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:37.678126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea7df4e-3a3d-40d2-9d30-4cb054c3770d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:37.678126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea7df4e-3a3d-40d2-9d30-4cb054c3770d None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:37.679451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 586/1178] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:37.684411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd979eda-6625-4cdd-b161-01fdd352bf4c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:37.684846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd979eda-6625-4cdd-b161-01fdd352bf4c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:37.684996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd979eda-6625-4cdd-b161-01fdd352bf4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:37.701612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-cd979eda-6625-4cdd-b161-01fdd352bf4c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:37.702781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-cd979eda-6625-4cdd-b161-01fdd352bf4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:37.766243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 593/1179] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:37 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:47:37.789265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea7df4e-3a3d-40d2-9d30-4cb054c3770d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:37.907639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea7df4e-3a3d-40d2-9d30-4cb054c3770d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:37.940124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d46f9a-b732-4ae4-9e51-78ebe01d6ec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:37.940873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d46f9a-b732-4ae4-9e51-78ebe01d6ec5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:37.941155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d46f9a-b732-4ae4-9e51-78ebe01d6ec5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:37.941427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74d46f9a-b732-4ae4-9e51-78ebe01d6ec5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:37.951035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74d46f9a-b732-4ae4-9e51-78ebe01d6ec5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-74d46f9a-b732-4ae4-9e51-78ebe01d6ec5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:47:37.958450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:47:37.959574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:47:37.960559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 587/1180] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:47:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:47:37.965645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd979eda-6625-4cdd-b161-01fdd352bf4c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:37.967767 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:38.062388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd979eda-6625-4cdd-b161-01fdd352bf4c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:38.094990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2466bf2b-8d2c-477b-9857-68d4195381ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:38.095638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2466bf2b-8d2c-477b-9857-68d4195381ba None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:38.095922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2466bf2b-8d2c-477b-9857-68d4195381ba None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:38.096196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2466bf2b-8d2c-477b-9857-68d4195381ba None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:38.096964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2466bf2b-8d2c-477b-9857-68d4195381ba None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:38.097270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2466bf2b-8d2c-477b-9857-68d4195381ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:38.098637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2466bf2b-8d2c-477b-9857-68d4195381ba None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:38.106782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 594/1181] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:47:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:38.113058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d46f9a-b732-4ae4-9e51-78ebe01d6ec5 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:38.136907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:38.216877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74d46f9a-b732-4ae4-9e51-78ebe01d6ec5 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:38.249485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd8399e-c2b6-491b-94f0-5f561ab2b9d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:38.250344 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd8399e-c2b6-491b-94f0-5f561ab2b9d5 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:38.250654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd8399e-c2b6-491b-94f0-5f561ab2b9d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:38.250936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dd8399e-c2b6-491b-94f0-5f561ab2b9d5 None admin] PATH_INFO: `/v3/users/f4c3c3425a004c73bbfe7f7d464b0ed9` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:38.263408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd8399e-c2b6-491b-94f0-5f561ab2b9d5 None admin] RBAC: Authorizing `identity:get_user(user_id=f4c3c3425a004c73bbfe7f7d464b0ed9)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:38.264881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dd8399e-c2b6-491b-94f0-5f561ab2b9d5 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:38.266555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 588/1182] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:38 2022] GET /identity/v3/users/f4c3c3425a004c73bbfe7f7d464b0ed9 => generated 311 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:38.272656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2466bf2b-8d2c-477b-9857-68d4195381ba None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:38.331268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:38.367716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2466bf2b-8d2c-477b-9857-68d4195381ba None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:38.399856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259ad63e-7d4c-4729-b390-9b79b04ad6e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:38.400481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259ad63e-7d4c-4729-b390-9b79b04ad6e3 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:38.400786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259ad63e-7d4c-4729-b390-9b79b04ad6e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:38.401065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259ad63e-7d4c-4729-b390-9b79b04ad6e3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:38.464843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259ad63e-7d4c-4729-b390-9b79b04ad6e3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:38.467683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259ad63e-7d4c-4729-b390-9b79b04ad6e3 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:38.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:38.565752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 595/1183] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:47:38 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:47:38.571647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd8399e-c2b6-491b-94f0-5f561ab2b9d5 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:38.648117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:38.666044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dd8399e-c2b6-491b-94f0-5f561ab2b9d5 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:38.699356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799ef65b-df45-48d3-8b4b-aa4f2a51f3e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:38.700200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799ef65b-df45-48d3-8b4b-aa4f2a51f3e8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:38.700483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799ef65b-df45-48d3-8b4b-aa4f2a51f3e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:38.700758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799ef65b-df45-48d3-8b4b-aa4f2a51f3e8 None admin] PATH_INFO: `/v3/projects/cd621d04d8ab47ec9af5dad2cd97c7f1` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:38.701939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799ef65b-df45-48d3-8b4b-aa4f2a51f3e8 None admin] RBAC: Authorizing `identity:get_project(project_id=cd621d04d8ab47ec9af5dad2cd97c7f1)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:38.703373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799ef65b-df45-48d3-8b4b-aa4f2a51f3e8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:38.705113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 589/1184] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:47:38 2022] GET /identity/v3/projects/cd621d04d8ab47ec9af5dad2cd97c7f1 => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:38.711101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259ad63e-7d4c-4729-b390-9b79b04ad6e3 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:38.806077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259ad63e-7d4c-4729-b390-9b79b04ad6e3 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:38.838531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9bae0ba-f60c-44f5-90df-61952667900a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:38.839172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bae0ba-f60c-44f5-90df-61952667900a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:38.839457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bae0ba-f60c-44f5-90df-61952667900a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:38.839738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9bae0ba-f60c-44f5-90df-61952667900a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:38.840558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bae0ba-f60c-44f5-90df-61952667900a None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=cd621d04d8ab47ec9af5dad2cd97c7f1, user.id=f4c3c3425a004c73bbfe7f7d464b0ed9) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:38.840901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bae0ba-f60c-44f5-90df-61952667900a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:38.842325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9bae0ba-f60c-44f5-90df-61952667900a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:38.848081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 596/1185] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:47:38 2022] GET /identity/v3/role_assignments?user.id=f4c3c3425a004c73bbfe7f7d464b0ed9&scope.project.id=cd621d04d8ab47ec9af5dad2cd97c7f1&role.id=766799d1b6c049168590571f1a959b3b => generated 259 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:40.942091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5bbb6f-2f48-4fce-ad66-baaf1859dcfc None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:40.942620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5bbb6f-2f48-4fce-ad66-baaf1859dcfc None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:40.942742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5bbb6f-2f48-4fce-ad66-baaf1859dcfc None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:40.944006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 590/1186] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:40.949220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8824fc25-da04-4980-b9e9-015232d898ea None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:40.949613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8824fc25-da04-4980-b9e9-015232d898ea None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:40.949778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8824fc25-da04-4980-b9e9-015232d898ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:40.966445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-8824fc25-da04-4980-b9e9-015232d898ea None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:40.967606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8824fc25-da04-4980-b9e9-015232d898ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:40.989445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:41.030985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 597/1187] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:40 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:41.063723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5bbb6f-2f48-4fce-ad66-baaf1859dcfc None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:41.161542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5bbb6f-2f48-4fce-ad66-baaf1859dcfc None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:41.193794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f8b2ca-fb12-433d-beb9-d945f8711b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:41.194508 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f8b2ca-fb12-433d-beb9-d945f8711b5a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:41.194793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f8b2ca-fb12-433d-beb9-d945f8711b5a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:41.195067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f8b2ca-fb12-433d-beb9-d945f8711b5a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:41.205333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f8b2ca-fb12-433d-beb9-d945f8711b5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-a5f8b2ca-fb12-433d-beb9-d945f8711b5a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:47:41.212914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:47:41.213978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:47:41.214837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 591/1188] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:47:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:47:41.220032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8824fc25-da04-4980-b9e9-015232d898ea None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:41.314353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8824fc25-da04-4980-b9e9-015232d898ea None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:41.351556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceda866a-8bad-47fc-bb78-eab76ceda20a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:41.352223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceda866a-8bad-47fc-bb78-eab76ceda20a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:41.352508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceda866a-8bad-47fc-bb78-eab76ceda20a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:41.352804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceda866a-8bad-47fc-bb78-eab76ceda20a None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:41.353554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceda866a-8bad-47fc-bb78-eab76ceda20a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:41.353863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceda866a-8bad-47fc-bb78-eab76ceda20a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:41.355229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceda866a-8bad-47fc-bb78-eab76ceda20a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:41.363047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 598/1189] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:47:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:41.368673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f8b2ca-fb12-433d-beb9-d945f8711b5a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:41.380249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:41.466476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f8b2ca-fb12-433d-beb9-d945f8711b5a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:41.497852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516f15fc-52e2-4be7-b0ab-1e663a2f647a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:41.498704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516f15fc-52e2-4be7-b0ab-1e663a2f647a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:41.498983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516f15fc-52e2-4be7-b0ab-1e663a2f647a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:41.499255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516f15fc-52e2-4be7-b0ab-1e663a2f647a None admin] PATH_INFO: `/v3/users/f4c3c3425a004c73bbfe7f7d464b0ed9` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:41.501065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516f15fc-52e2-4be7-b0ab-1e663a2f647a None admin] RBAC: Authorizing `identity:get_user(user_id=f4c3c3425a004c73bbfe7f7d464b0ed9)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:41.502446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516f15fc-52e2-4be7-b0ab-1e663a2f647a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:41.504048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 592/1190] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:41 2022] GET /identity/v3/users/f4c3c3425a004c73bbfe7f7d464b0ed9 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:41.509721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceda866a-8bad-47fc-bb78-eab76ceda20a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:41.531096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:41.604015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceda866a-8bad-47fc-bb78-eab76ceda20a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:41.636823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e80ca7-5d44-4900-a3c3-8485e535aac0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:41.637665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e80ca7-5d44-4900-a3c3-8485e535aac0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:41.637948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e80ca7-5d44-4900-a3c3-8485e535aac0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:41.638223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e80ca7-5d44-4900-a3c3-8485e535aac0 None admin] PATH_INFO: `/v3/projects/cd621d04d8ab47ec9af5dad2cd97c7f1` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:41.639381 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e80ca7-5d44-4900-a3c3-8485e535aac0 None admin] RBAC: Authorizing `identity:get_project(project_id=cd621d04d8ab47ec9af5dad2cd97c7f1)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:41.640827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e80ca7-5d44-4900-a3c3-8485e535aac0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:41.642500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 599/1191] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:47:41 2022] GET /identity/v3/projects/cd621d04d8ab47ec9af5dad2cd97c7f1 => generated 361 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:41.648426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516f15fc-52e2-4be7-b0ab-1e663a2f647a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:41.723506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:41.745778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516f15fc-52e2-4be7-b0ab-1e663a2f647a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:41.778660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60e2a11c-6875-4c89-bda2-36c3ca28259f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:41.779529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e2a11c-6875-4c89-bda2-36c3ca28259f None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:41.779813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e2a11c-6875-4c89-bda2-36c3ca28259f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:41.780091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60e2a11c-6875-4c89-bda2-36c3ca28259f None admin] PATH_INFO: `/v3/projects/cd621d04d8ab47ec9af5dad2cd97c7f1/users/f4c3c3425a004c73bbfe7f7d464b0ed9/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:41.782613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e2a11c-6875-4c89-bda2-36c3ca28259f None admin] RBAC: Authorizing `identity:create_grant(project_id=cd621d04d8ab47ec9af5dad2cd97c7f1, user_id=f4c3c3425a004c73bbfe7f7d464b0ed9, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:41.784038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60e2a11c-6875-4c89-bda2-36c3ca28259f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:41.846318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 593/1192] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:47:41 2022] PUT /identity/v3/projects/cd621d04d8ab47ec9af5dad2cd97c7f1/users/f4c3c3425a004c73bbfe7f7d464b0ed9/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:47:43.911611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7175a3-3ec8-453e-95c6-8cee34e7abd9 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:43.912167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7175a3-3ec8-453e-95c6-8cee34e7abd9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:43.912167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7175a3-3ec8-453e-95c6-8cee34e7abd9 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:43.913560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 600/1193] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:47:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:47:43.918708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280c511-53a7-49ed-aaf0-6d1742126b9b None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:43.919114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280c511-53a7-49ed-aaf0-6d1742126b9b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:43.919263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5280c511-53a7-49ed-aaf0-6d1742126b9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:43.936078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5280c511-53a7-49ed-aaf0-6d1742126b9b None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:47:43.937292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5280c511-53a7-49ed-aaf0-6d1742126b9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:47:44.001332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 594/1194] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:47:43 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:47:44.024409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7175a3-3ec8-453e-95c6-8cee34e7abd9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:44.111005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:44.123117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7175a3-3ec8-453e-95c6-8cee34e7abd9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:44.156343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3607f883-6df5-4659-938c-5c0055aa2e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:44.157189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3607f883-6df5-4659-938c-5c0055aa2e29 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:44.157470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3607f883-6df5-4659-938c-5c0055aa2e29 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:44.157743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3607f883-6df5-4659-938c-5c0055aa2e29 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:44.167292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3607f883-6df5-4659-938c-5c0055aa2e29 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-3607f883-6df5-4659-938c-5c0055aa2e29 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:47:44.174676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:47:44.175759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:47:44.176863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 601/1195] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:47:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:47:44.181837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5280c511-53a7-49ed-aaf0-6d1742126b9b None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:44.243645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:44.277442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5280c511-53a7-49ed-aaf0-6d1742126b9b None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:44.309946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067bf816-f0ef-4ef6-a551-16e0e07b4a7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:44.310595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067bf816-f0ef-4ef6-a551-16e0e07b4a7a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:44.310877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067bf816-f0ef-4ef6-a551-16e0e07b4a7a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:44.311154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067bf816-f0ef-4ef6-a551-16e0e07b4a7a None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:44.311891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067bf816-f0ef-4ef6-a551-16e0e07b4a7a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:44.312198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067bf816-f0ef-4ef6-a551-16e0e07b4a7a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:44.314116 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067bf816-f0ef-4ef6-a551-16e0e07b4a7a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:44.322011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 595/1196] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:47:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:44.327575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3607f883-6df5-4659-938c-5c0055aa2e29 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:44.423618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3607f883-6df5-4659-938c-5c0055aa2e29 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:44.455155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec9ae6-c872-4f79-b40d-d2aa976be2c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:44.456000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec9ae6-c872-4f79-b40d-d2aa976be2c1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:44.456279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec9ae6-c872-4f79-b40d-d2aa976be2c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:44.456551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec9ae6-c872-4f79-b40d-d2aa976be2c1 None admin] PATH_INFO: `/v3/users/f4c3c3425a004c73bbfe7f7d464b0ed9` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:44.457723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ec9ae6-c872-4f79-b40d-d2aa976be2c1 None admin] RBAC: Authorizing `identity:get_user(user_id=f4c3c3425a004c73bbfe7f7d464b0ed9)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:44.459152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ec9ae6-c872-4f79-b40d-d2aa976be2c1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:44.460751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 602/1197] 10.222.0.25 () {56 vars in 1177 bytes} [Fri Jun 10 12:47:44 2022] GET /identity/v3/users/f4c3c3425a004c73bbfe7f7d464b0ed9 => generated 311 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:44.466477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067bf816-f0ef-4ef6-a551-16e0e07b4a7a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:44.562029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067bf816-f0ef-4ef6-a551-16e0e07b4a7a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:44.586823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:44.594211 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219c5cff-6dd9-4782-bdcf-7e85b9700242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:44.595889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219c5cff-6dd9-4782-bdcf-7e85b9700242 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:44.595889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219c5cff-6dd9-4782-bdcf-7e85b9700242 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:44.595889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219c5cff-6dd9-4782-bdcf-7e85b9700242 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:44.659280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219c5cff-6dd9-4782-bdcf-7e85b9700242 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:44.660711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219c5cff-6dd9-4782-bdcf-7e85b9700242 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:44.752823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 596/1198] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:47:44 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:47:44.758495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec9ae6-c872-4f79-b40d-d2aa976be2c1 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:44.777074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:44.854149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec9ae6-c872-4f79-b40d-d2aa976be2c1 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:44.886027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2180832-316a-41ba-aeb0-4907e23f12d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:44.886848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2180832-316a-41ba-aeb0-4907e23f12d7 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:44.887146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2180832-316a-41ba-aeb0-4907e23f12d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:44.887397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2180832-316a-41ba-aeb0-4907e23f12d7 None admin] PATH_INFO: `/v3/projects/cd621d04d8ab47ec9af5dad2cd97c7f1` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:44.888549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2180832-316a-41ba-aeb0-4907e23f12d7 None admin] RBAC: Authorizing `identity:get_project(project_id=cd621d04d8ab47ec9af5dad2cd97c7f1)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:44.889992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2180832-316a-41ba-aeb0-4907e23f12d7 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:44.891658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 603/1199] 10.222.0.25 () {56 vars in 1186 bytes} [Fri Jun 10 12:47:44 2022] GET /identity/v3/projects/cd621d04d8ab47ec9af5dad2cd97c7f1 => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:44.898078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219c5cff-6dd9-4782-bdcf-7e85b9700242 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:47:44.925350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:44.993503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219c5cff-6dd9-4782-bdcf-7e85b9700242 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:47:45.025995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1af235-c4c6-4455-a4ae-1cf294563999 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:47:45.026579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1af235-c4c6-4455-a4ae-1cf294563999 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:47:45.026865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1af235-c4c6-4455-a4ae-1cf294563999 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:47:45.027144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1af235-c4c6-4455-a4ae-1cf294563999 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:47:45.027960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1af235-c4c6-4455-a4ae-1cf294563999 None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=cd621d04d8ab47ec9af5dad2cd97c7f1, user.id=f4c3c3425a004c73bbfe7f7d464b0ed9) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:47:45.028271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1af235-c4c6-4455-a4ae-1cf294563999 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:47:45.029723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1af235-c4c6-4455-a4ae-1cf294563999 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:47:45.036022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 597/1200] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:47:44 2022] GET /identity/v3/role_assignments?user.id=f4c3c3425a004c73bbfe7f7d464b0ed9&scope.project.id=cd621d04d8ab47ec9af5dad2cd97c7f1&role.id=766799d1b6c049168590571f1a959b3b => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:47:54.882076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=58 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:47:55.033372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=54 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:01.406826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab30cc6-419b-4b30-bdbb-f92d5a6d4ef5 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:01.407792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab30cc6-419b-4b30-bdbb-f92d5a6d4ef5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:01.408120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab30cc6-419b-4b30-bdbb-f92d5a6d4ef5 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:01.409645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 604/1201] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:01.415080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ceb74d-1632-4c49-b8ef-99ae73c92c30 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:01.415540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ceb74d-1632-4c49-b8ef-99ae73c92c30 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:01.415676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89ceb74d-1632-4c49-b8ef-99ae73c92c30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:01.432815 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-89ceb74d-1632-4c49-b8ef-99ae73c92c30 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:01.434019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-89ceb74d-1632-4c49-b8ef-99ae73c92c30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:01.498091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 598/1202] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:01 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:01.524050 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab30cc6-419b-4b30-bdbb-f92d5a6d4ef5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:01.625090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab30cc6-419b-4b30-bdbb-f92d5a6d4ef5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:01.657449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1412f7b6-a347-4974-855e-cb4e3f2eefb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:01.658218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1412f7b6-a347-4974-855e-cb4e3f2eefb9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:01.658495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1412f7b6-a347-4974-855e-cb4e3f2eefb9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:01.658764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1412f7b6-a347-4974-855e-cb4e3f2eefb9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:01.668496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1412f7b6-a347-4974-855e-cb4e3f2eefb9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:01.670086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1412f7b6-a347-4974-855e-cb4e3f2eefb9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-1412f7b6-a347-4974-855e-cb4e3f2eefb9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:01.688059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:01.689225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:01.690404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:01.690404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:01.690404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:01.690404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:01.690404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 605/1203] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:01.695477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ceb74d-1632-4c49-b8ef-99ae73c92c30 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:01.791347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89ceb74d-1632-4c49-b8ef-99ae73c92c30 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:01.819664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:01.825132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-127fb56a-7f1c-4b6b-8517-7d37fbe100ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:01.825782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127fb56a-7f1c-4b6b-8517-7d37fbe100ab None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:01.826074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127fb56a-7f1c-4b6b-8517-7d37fbe100ab None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:01.826378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-127fb56a-7f1c-4b6b-8517-7d37fbe100ab None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:01.827112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127fb56a-7f1c-4b6b-8517-7d37fbe100ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:01.827417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127fb56a-7f1c-4b6b-8517-7d37fbe100ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:01.828803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-127fb56a-7f1c-4b6b-8517-7d37fbe100ab None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:01.839754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 599/1204] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:01.845574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1412f7b6-a347-4974-855e-cb4e3f2eefb9 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:01.897452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:01.947292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1412f7b6-a347-4974-855e-cb4e3f2eefb9 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:01.978815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a6a3630-0cd6-41de-a949-2759d9643e25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:01.979474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6a3630-0cd6-41de-a949-2759d9643e25 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:01.979762 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6a3630-0cd6-41de-a949-2759d9643e25 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:01.980046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a6a3630-0cd6-41de-a949-2759d9643e25 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:01.980745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6a3630-0cd6-41de-a949-2759d9643e25 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:01.982213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a6a3630-0cd6-41de-a949-2759d9643e25 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:02.065786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 606/1205] 10.222.0.25 () {60 vars in 1132 bytes} [Fri Jun 10 12:49:01 2022] POST /identity/v3/users => generated 251 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:49:04.196329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb8e524-bc49-47eb-8ac6-e52bae84ca56 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:04.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb8e524-bc49-47eb-8ac6-e52bae84ca56 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:04.196941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb8e524-bc49-47eb-8ac6-e52bae84ca56 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:04.198306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 600/1206] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:04.203443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909d3505-e04e-46af-aff3-0f39feec7377 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:04.203860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909d3505-e04e-46af-aff3-0f39feec7377 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:04.204006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-909d3505-e04e-46af-aff3-0f39feec7377 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:04.220407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-909d3505-e04e-46af-aff3-0f39feec7377 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:04.229262 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-909d3505-e04e-46af-aff3-0f39feec7377 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:04.286637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 607/1207] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:04 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:04.310837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb8e524-bc49-47eb-8ac6-e52bae84ca56 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:04.421750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb8e524-bc49-47eb-8ac6-e52bae84ca56 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:04.461231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7395605c-0ac3-4319-831c-2e9439679ac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:04.461949 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7395605c-0ac3-4319-831c-2e9439679ac7 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:04.462342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7395605c-0ac3-4319-831c-2e9439679ac7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:04.462628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7395605c-0ac3-4319-831c-2e9439679ac7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:04.473108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7395605c-0ac3-4319-831c-2e9439679ac7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-7395605c-0ac3-4319-831c-2e9439679ac7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:49:04.480726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:04.481923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:04.482910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 601/1208] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:49:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:04.487876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909d3505-e04e-46af-aff3-0f39feec7377 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:04.527951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:04.584032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-909d3505-e04e-46af-aff3-0f39feec7377 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:04.616904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f4e6cf-f8d4-4750-bb7c-22accbe8516a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:04.617555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f4e6cf-f8d4-4750-bb7c-22accbe8516a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:04.617836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f4e6cf-f8d4-4750-bb7c-22accbe8516a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:04.618110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f4e6cf-f8d4-4750-bb7c-22accbe8516a None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:04.618840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f4e6cf-f8d4-4750-bb7c-22accbe8516a None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:04.619141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f4e6cf-f8d4-4750-bb7c-22accbe8516a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:04.620502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f4e6cf-f8d4-4750-bb7c-22accbe8516a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:04.628578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 608/1209] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:49:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:04.634343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7395605c-0ac3-4319-831c-2e9439679ac7 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:04.688318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:04.736424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7395605c-0ac3-4319-831c-2e9439679ac7 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:04.769739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94cda85-91de-4b38-8abb-359fafa90724 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:04.770520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94cda85-91de-4b38-8abb-359fafa90724 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:04.770802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94cda85-91de-4b38-8abb-359fafa90724 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:04.771077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94cda85-91de-4b38-8abb-359fafa90724 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:04.781546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94cda85-91de-4b38-8abb-359fafa90724 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:04.783012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94cda85-91de-4b38-8abb-359fafa90724 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-b94cda85-91de-4b38-8abb-359fafa90724 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:04.800314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:04.801462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:04.802584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:04.802584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:04.802584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:04.802584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:04.802584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 602/1210] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:04.807390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f4e6cf-f8d4-4750-bb7c-22accbe8516a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:04.900504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:04.903483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f4e6cf-f8d4-4750-bb7c-22accbe8516a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:04.935167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea0e52c-ad5a-4b4e-928e-49ad74bf9692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:04.936332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0e52c-ad5a-4b4e-928e-49ad74bf9692 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:04.936615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0e52c-ad5a-4b4e-928e-49ad74bf9692 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:04.936920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea0e52c-ad5a-4b4e-928e-49ad74bf9692 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:04.937626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea0e52c-ad5a-4b4e-928e-49ad74bf9692 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:04.937965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea0e52c-ad5a-4b4e-928e-49ad74bf9692 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:04.939344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea0e52c-ad5a-4b4e-928e-49ad74bf9692 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:04.950105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 609/1211] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:04.955630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94cda85-91de-4b38-8abb-359fafa90724 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:05.057437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94cda85-91de-4b38-8abb-359fafa90724 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:05.059159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:05.092501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a510a0aa-9b4d-4d25-bd89-5423a15fddb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:05.093319 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a510a0aa-9b4d-4d25-bd89-5423a15fddb1 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:05.093600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a510a0aa-9b4d-4d25-bd89-5423a15fddb1 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:05.093872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a510a0aa-9b4d-4d25-bd89-5423a15fddb1 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:05.102209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a510a0aa-9b4d-4d25-bd89-5423a15fddb1 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:05.103706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a510a0aa-9b4d-4d25-bd89-5423a15fddb1 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-a510a0aa-9b4d-4d25-bd89-5423a15fddb1 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:49:05.117574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:05.118746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:05.119971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 603/1212] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:04 2022] GET /identity/v3/users/placement => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:05.127440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea0e52c-ad5a-4b4e-928e-49ad74bf9692 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:05.220836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea0e52c-ad5a-4b4e-928e-49ad74bf9692 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:05.252797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba933a8-aafb-454f-b420-f836692bb552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:05.253445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba933a8-aafb-454f-b420-f836692bb552 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:05.253726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba933a8-aafb-454f-b420-f836692bb552 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:05.254001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba933a8-aafb-454f-b420-f836692bb552 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:05.254823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba933a8-aafb-454f-b420-f836692bb552 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:05.255130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba933a8-aafb-454f-b420-f836692bb552 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:05.256555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba933a8-aafb-454f-b420-f836692bb552 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:05.273778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 610/1213] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Jun 10 12:49:05 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:05.279416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a510a0aa-9b4d-4d25-bd89-5423a15fddb1 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:05.375682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a510a0aa-9b4d-4d25-bd89-5423a15fddb1 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:05.407179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ad90e-2647-494e-b5e2-49473eb1efcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:05.407810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ad90e-2647-494e-b5e2-49473eb1efcb None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:05.408091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ad90e-2647-494e-b5e2-49473eb1efcb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:05.408415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70ad90e-2647-494e-b5e2-49473eb1efcb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:05.437608 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:05.472942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ad90e-2647-494e-b5e2-49473eb1efcb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:05.474422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f70ad90e-2647-494e-b5e2-49473eb1efcb None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:05.568889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 604/1214] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:49:05 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:49:05.574602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba933a8-aafb-454f-b420-f836692bb552 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:05.592623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:05.671230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba933a8-aafb-454f-b420-f836692bb552 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:05.704734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb545483-62d5-4ef1-9a3f-ddec56061bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:05.705539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb545483-62d5-4ef1-9a3f-ddec56061bc0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:05.705821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb545483-62d5-4ef1-9a3f-ddec56061bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:05.706097 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb545483-62d5-4ef1-9a3f-ddec56061bc0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:05.715660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb545483-62d5-4ef1-9a3f-ddec56061bc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:05.717673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb545483-62d5-4ef1-9a3f-ddec56061bc0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-fb545483-62d5-4ef1-9a3f-ddec56061bc0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:05.734284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:05.735343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:05.736602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:05.736602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:05.736602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:05.736602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:05.736602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 611/1215] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:05.741122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ad90e-2647-494e-b5e2-49473eb1efcb None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:05.801153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:05.838540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70ad90e-2647-494e-b5e2-49473eb1efcb None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:05.871270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049afc2f-5710-4ee2-89d2-9bde855f457e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:05.871842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049afc2f-5710-4ee2-89d2-9bde855f457e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:05.872128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049afc2f-5710-4ee2-89d2-9bde855f457e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:05.872409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049afc2f-5710-4ee2-89d2-9bde855f457e None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:05.873673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049afc2f-5710-4ee2-89d2-9bde855f457e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:05.873998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049afc2f-5710-4ee2-89d2-9bde855f457e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:05.875361 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049afc2f-5710-4ee2-89d2-9bde855f457e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:05.886529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 605/1216] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:05.892716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb545483-62d5-4ef1-9a3f-ddec56061bc0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:05.958181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:05.986948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb545483-62d5-4ef1-9a3f-ddec56061bc0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:06.020303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6520b68-d01c-44d2-88a9-db8ed0929bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:06.021107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6520b68-d01c-44d2-88a9-db8ed0929bc0 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:06.021400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6520b68-d01c-44d2-88a9-db8ed0929bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:06.021678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6520b68-d01c-44d2-88a9-db8ed0929bc0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:06.028079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6520b68-d01c-44d2-88a9-db8ed0929bc0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:06.029594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6520b68-d01c-44d2-88a9-db8ed0929bc0 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-b6520b68-d01c-44d2-88a9-db8ed0929bc0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:49:06.043486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:06.044615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:06.045709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:49:06.045709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:49:06.045709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:49:06.045709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:49:06.045709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:49:06.045709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:49:06.045709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:06.045709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:06.045709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 612/1217] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:49:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:06.050492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049afc2f-5710-4ee2-89d2-9bde855f457e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:06.148540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049afc2f-5710-4ee2-89d2-9bde855f457e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:06.169486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:06.180930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51af619b-7b5e-45cc-a2b1-f825fa0a9db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:06.181572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51af619b-7b5e-45cc-a2b1-f825fa0a9db3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:06.181855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51af619b-7b5e-45cc-a2b1-f825fa0a9db3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:06.182129 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51af619b-7b5e-45cc-a2b1-f825fa0a9db3 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:06.182904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51af619b-7b5e-45cc-a2b1-f825fa0a9db3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:06.183255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51af619b-7b5e-45cc-a2b1-f825fa0a9db3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:06.184685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51af619b-7b5e-45cc-a2b1-f825fa0a9db3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:06.196776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 606/1218] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:49:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:06.202938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6520b68-d01c-44d2-88a9-db8ed0929bc0 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:06.299667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6520b68-d01c-44d2-88a9-db8ed0929bc0 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:06.328074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:06.333796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc99c4df-23e6-495f-8fd4-74e77008e5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:06.334449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc99c4df-23e6-495f-8fd4-74e77008e5ec None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:06.334734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc99c4df-23e6-495f-8fd4-74e77008e5ec None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:06.335010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc99c4df-23e6-495f-8fd4-74e77008e5ec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:06.335832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc99c4df-23e6-495f-8fd4-74e77008e5ec None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=0c315cfb1aa44f3c9a755f3cb8e5cd5d) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:06.336144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc99c4df-23e6-495f-8fd4-74e77008e5ec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:06.337634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc99c4df-23e6-495f-8fd4-74e77008e5ec None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:06.342919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 613/1219] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:49:06 2022] GET /identity/v3/role_assignments?user.id=0c315cfb1aa44f3c9a755f3cb8e5cd5d&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:08.423395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d838ef37-b81f-4fc2-a681-28c6232f1aa0 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:08.423951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d838ef37-b81f-4fc2-a681-28c6232f1aa0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:08.423951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d838ef37-b81f-4fc2-a681-28c6232f1aa0 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:08.425371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 607/1220] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:08.430732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4471d2c-cb9a-4157-a168-2799c1e0063e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:08.431180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4471d2c-cb9a-4157-a168-2799c1e0063e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:08.431288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4471d2c-cb9a-4157-a168-2799c1e0063e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:08.448484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-f4471d2c-cb9a-4157-a168-2799c1e0063e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:08.449679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f4471d2c-cb9a-4157-a168-2799c1e0063e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:08.514603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 614/1221] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:08 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:08.538018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d838ef37-b81f-4fc2-a681-28c6232f1aa0 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:08.637024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d838ef37-b81f-4fc2-a681-28c6232f1aa0 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:08.672399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2c83bb-4310-45b1-a83a-d2be9c16b7d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:08.673225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2c83bb-4310-45b1-a83a-d2be9c16b7d4 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:08.673515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2c83bb-4310-45b1-a83a-d2be9c16b7d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:08.673795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2c83bb-4310-45b1-a83a-d2be9c16b7d4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:08.684004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2c83bb-4310-45b1-a83a-d2be9c16b7d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-ee2c83bb-4310-45b1-a83a-d2be9c16b7d4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:49:08.692214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:08.693448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:08.694483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 608/1222] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:49:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:08.699578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4471d2c-cb9a-4157-a168-2799c1e0063e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:08.799030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4471d2c-cb9a-4157-a168-2799c1e0063e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:08.821090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:08.831486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870cd034-3921-487c-bad1-7e7d0792e5e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:08.832138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870cd034-3921-487c-bad1-7e7d0792e5e6 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:08.832418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870cd034-3921-487c-bad1-7e7d0792e5e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:08.832690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870cd034-3921-487c-bad1-7e7d0792e5e6 None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:08.833456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870cd034-3921-487c-bad1-7e7d0792e5e6 None admin] RBAC: Adding query filter params (name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:08.833813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870cd034-3921-487c-bad1-7e7d0792e5e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:08.835190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870cd034-3921-487c-bad1-7e7d0792e5e6 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:08.843219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 615/1223] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:49:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:08.849081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2c83bb-4310-45b1-a83a-d2be9c16b7d4 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:08.891209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:08.945476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2c83bb-4310-45b1-a83a-d2be9c16b7d4 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:08.977046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e752220a-494b-40d8-a5ca-aae025a43744 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:08.977818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e752220a-494b-40d8-a5ca-aae025a43744 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:08.978102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e752220a-494b-40d8-a5ca-aae025a43744 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:08.978379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e752220a-494b-40d8-a5ca-aae025a43744 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:08.989728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e752220a-494b-40d8-a5ca-aae025a43744 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:08.994241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e752220a-494b-40d8-a5ca-aae025a43744 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-e752220a-494b-40d8-a5ca-aae025a43744 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:09.007683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:09.008814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:09.009789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:09.009789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:09.009789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:09.009789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:09.009789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 609/1224] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:09.014833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870cd034-3921-487c-bad1-7e7d0792e5e6 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:09.113493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870cd034-3921-487c-bad1-7e7d0792e5e6 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:09.146019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d857584-d53c-40c6-827f-3d8301866418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:09.146610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d857584-d53c-40c6-827f-3d8301866418 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:09.146896 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d857584-d53c-40c6-827f-3d8301866418 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:09.147175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d857584-d53c-40c6-827f-3d8301866418 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:09.147886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d857584-d53c-40c6-827f-3d8301866418 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:09.148195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d857584-d53c-40c6-827f-3d8301866418 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:09.149598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d857584-d53c-40c6-827f-3d8301866418 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:09.161930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 616/1225] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:09.167804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e752220a-494b-40d8-a5ca-aae025a43744 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:09.260714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:09.267446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e752220a-494b-40d8-a5ca-aae025a43744 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:09.299737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da00ddb3-e80a-4ba8-9407-c22df937f291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:09.300542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da00ddb3-e80a-4ba8-9407-c22df937f291 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:09.300831 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da00ddb3-e80a-4ba8-9407-c22df937f291 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:09.301107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da00ddb3-e80a-4ba8-9407-c22df937f291 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:09.308132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da00ddb3-e80a-4ba8-9407-c22df937f291 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:09.309647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da00ddb3-e80a-4ba8-9407-c22df937f291 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-da00ddb3-e80a-4ba8-9407-c22df937f291 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:49:09.324074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:09.325237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:09.326599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 610/1226] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:09 2022] GET /identity/v3/users/placement => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:09.331102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d857584-d53c-40c6-827f-3d8301866418 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:09.380379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:09.431547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d857584-d53c-40c6-827f-3d8301866418 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:09.463275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8621e23-9a63-4cbe-9f86-2846a00cfe9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:09.463924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8621e23-9a63-4cbe-9f86-2846a00cfe9f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:09.464203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8621e23-9a63-4cbe-9f86-2846a00cfe9f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:09.464477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8621e23-9a63-4cbe-9f86-2846a00cfe9f None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:09.465329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8621e23-9a63-4cbe-9f86-2846a00cfe9f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:09.465634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8621e23-9a63-4cbe-9f86-2846a00cfe9f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:09.467074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8621e23-9a63-4cbe-9f86-2846a00cfe9f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:09.483968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 617/1227] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Jun 10 12:49:09 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:49:09.489589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da00ddb3-e80a-4ba8-9407-c22df937f291 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:09.588132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da00ddb3-e80a-4ba8-9407-c22df937f291 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:09.621589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f478425-9451-4a7f-a21b-5886d18cd796 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:09.622370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f478425-9451-4a7f-a21b-5886d18cd796 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:09.624232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f478425-9451-4a7f-a21b-5886d18cd796 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:09.624232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f478425-9451-4a7f-a21b-5886d18cd796 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:09.633517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f478425-9451-4a7f-a21b-5886d18cd796 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:09.634972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f478425-9451-4a7f-a21b-5886d18cd796 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-8f478425-9451-4a7f-a21b-5886d18cd796 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:09.651792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:09.654743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:09.655654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:09.655654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:09.655654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:09.655654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:09.655654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 611/1228] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:09.659373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8621e23-9a63-4cbe-9f86-2846a00cfe9f None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:09.756332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8621e23-9a63-4cbe-9f86-2846a00cfe9f None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:09.758593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:09.790087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbd38c7-37f1-4f62-b642-458f5d9057cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:09.790737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd38c7-37f1-4f62-b642-458f5d9057cc None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:09.791016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd38c7-37f1-4f62-b642-458f5d9057cc None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:09.791287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbd38c7-37f1-4f62-b642-458f5d9057cc None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:09.791991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbd38c7-37f1-4f62-b642-458f5d9057cc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:09.792293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbd38c7-37f1-4f62-b642-458f5d9057cc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:09.793671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbd38c7-37f1-4f62-b642-458f5d9057cc None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:09.805205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 618/1229] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:09.810850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f478425-9451-4a7f-a21b-5886d18cd796 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:09.820243 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:09.908168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f478425-9451-4a7f-a21b-5886d18cd796 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:09.939821 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0aa0b8-4723-4ba5-88d0-a24339ae4242 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:09.940598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0aa0b8-4723-4ba5-88d0-a24339ae4242 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:09.940910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0aa0b8-4723-4ba5-88d0-a24339ae4242 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:09.941190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0aa0b8-4723-4ba5-88d0-a24339ae4242 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:09.947775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0aa0b8-4723-4ba5-88d0-a24339ae4242 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:09.949590 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0aa0b8-4723-4ba5-88d0-a24339ae4242 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-6f0aa0b8-4723-4ba5-88d0-a24339ae4242 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:49:09.964651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:09.965763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:09.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:49:09.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:49:09.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:49:09.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:49:09.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:49:09.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:49:09.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:09.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:09.971965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 612/1230] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:49:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:09.972645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbd38c7-37f1-4f62-b642-458f5d9057cc None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:10.069752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbd38c7-37f1-4f62-b642-458f5d9057cc None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:10.102751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032a5fcc-f1c0-4e35-a330-d9f97e67fe9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:10.103433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a5fcc-f1c0-4e35-a330-d9f97e67fe9a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:10.103734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a5fcc-f1c0-4e35-a330-d9f97e67fe9a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:10.104012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a5fcc-f1c0-4e35-a330-d9f97e67fe9a None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:10.104835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032a5fcc-f1c0-4e35-a330-d9f97e67fe9a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:10.105145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032a5fcc-f1c0-4e35-a330-d9f97e67fe9a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:10.106581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032a5fcc-f1c0-4e35-a330-d9f97e67fe9a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:10.118440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 619/1231] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:49:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:49:10.124445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0aa0b8-4723-4ba5-88d0-a24339ae4242 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:10.188556 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:10.222082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0aa0b8-4723-4ba5-88d0-a24339ae4242 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:10.254343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1debdcc6-6957-4d21-a4c2-486253db557f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:10.255208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1debdcc6-6957-4d21-a4c2-486253db557f None admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:10.255490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1debdcc6-6957-4d21-a4c2-486253db557f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:10.255768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1debdcc6-6957-4d21-a4c2-486253db557f None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/0c315cfb1aa44f3c9a755f3cb8e5cd5d/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:10.270252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1debdcc6-6957-4d21-a4c2-486253db557f None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=0c315cfb1aa44f3c9a755f3cb8e5cd5d, role_id=c7c592df9b6b4743a1ba6ca3b2a7fdbc)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:10.271694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1debdcc6-6957-4d21-a4c2-486253db557f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:10.361736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 613/1232] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:49:10 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/0c315cfb1aa44f3c9a755f3cb8e5cd5d/roles/c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 12:49:12.456521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c629d-9050-452e-ac41-b15b32cb2b2b None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:12.457087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c629d-9050-452e-ac41-b15b32cb2b2b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:12.457194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c629d-9050-452e-ac41-b15b32cb2b2b None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:12.458477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 620/1233] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:12.463709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9626724e-59de-452a-9f6e-86d1acbecc1d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:12.464114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9626724e-59de-452a-9f6e-86d1acbecc1d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:12.464273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9626724e-59de-452a-9f6e-86d1acbecc1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:12.480406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9626724e-59de-452a-9f6e-86d1acbecc1d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:12.481648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9626724e-59de-452a-9f6e-86d1acbecc1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:12.550480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 614/1234] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:12 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:12.573427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337c629d-9050-452e-ac41-b15b32cb2b2b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:12.599718 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:12.671351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337c629d-9050-452e-ac41-b15b32cb2b2b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:12.704305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abcebe1-2097-4ecf-8cb0-2f1e51590354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:12.705113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abcebe1-2097-4ecf-8cb0-2f1e51590354 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:12.705398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abcebe1-2097-4ecf-8cb0-2f1e51590354 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:12.705674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abcebe1-2097-4ecf-8cb0-2f1e51590354 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:12.715444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abcebe1-2097-4ecf-8cb0-2f1e51590354 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-1abcebe1-2097-4ecf-8cb0-2f1e51590354 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:49:12.723711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:12.724850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:12.725728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 621/1235] 10.222.0.25 () {56 vars in 1102 bytes} [Fri Jun 10 12:49:12 2022] GET /identity/v3/roles/service => generated 85 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:12.731051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9626724e-59de-452a-9f6e-86d1acbecc1d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:12.776511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:12.827901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9626724e-59de-452a-9f6e-86d1acbecc1d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:12.861535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f858000d-a7c3-46f9-b7d7-dfb33df9cb4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:12.862139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f858000d-a7c3-46f9-b7d7-dfb33df9cb4a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:12.862425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f858000d-a7c3-46f9-b7d7-dfb33df9cb4a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:12.862704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f858000d-a7c3-46f9-b7d7-dfb33df9cb4a None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:12.863445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f858000d-a7c3-46f9-b7d7-dfb33df9cb4a None admin] RBAC: Adding query filter params (name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:12.863757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f858000d-a7c3-46f9-b7d7-dfb33df9cb4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:12.865161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f858000d-a7c3-46f9-b7d7-dfb33df9cb4a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:12.873393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 615/1236] 10.222.0.25 () {56 vars in 1103 bytes} [Fri Jun 10 12:49:12 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:12.879151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abcebe1-2097-4ecf-8cb0-2f1e51590354 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:12.961893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:12.984725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abcebe1-2097-4ecf-8cb0-2f1e51590354 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:13.018749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919175a-5e3d-4d97-9fb0-7c030738d1e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:13.019575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919175a-5e3d-4d97-9fb0-7c030738d1e9 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:13.019861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919175a-5e3d-4d97-9fb0-7c030738d1e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:13.020137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c919175a-5e3d-4d97-9fb0-7c030738d1e9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:13.030765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c919175a-5e3d-4d97-9fb0-7c030738d1e9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:13.032209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c919175a-5e3d-4d97-9fb0-7c030738d1e9 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c919175a-5e3d-4d97-9fb0-7c030738d1e9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:13.050941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:13.053650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:13.054739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:13.054739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:13.054739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:13.054739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:13.054739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 622/1237] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:13.058113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f858000d-a7c3-46f9-b7d7-dfb33df9cb4a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:13.155874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f858000d-a7c3-46f9-b7d7-dfb33df9cb4a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:13.164560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:13.187845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab32db9-a0ad-4242-b93d-e5d1c5045fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:13.188552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab32db9-a0ad-4242-b93d-e5d1c5045fc8 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:13.188874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab32db9-a0ad-4242-b93d-e5d1c5045fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:13.189143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab32db9-a0ad-4242-b93d-e5d1c5045fc8 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:13.189843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab32db9-a0ad-4242-b93d-e5d1c5045fc8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:13.190145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab32db9-a0ad-4242-b93d-e5d1c5045fc8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:13.191500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab32db9-a0ad-4242-b93d-e5d1c5045fc8 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:13.202899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 616/1238] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:13.208469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919175a-5e3d-4d97-9fb0-7c030738d1e9 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:13.305538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c919175a-5e3d-4d97-9fb0-7c030738d1e9 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:13.343160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff85fabf-ef72-49c1-b5dd-d04557dae316 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:13.343959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff85fabf-ef72-49c1-b5dd-d04557dae316 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:13.344242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff85fabf-ef72-49c1-b5dd-d04557dae316 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:13.344515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff85fabf-ef72-49c1-b5dd-d04557dae316 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:13.347312 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:13.359181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff85fabf-ef72-49c1-b5dd-d04557dae316 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:13.360629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff85fabf-ef72-49c1-b5dd-d04557dae316 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-ff85fabf-ef72-49c1-b5dd-d04557dae316 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:49:13.375383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:13.376529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:13.377706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 623/1239] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:13 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:13.382657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab32db9-a0ad-4242-b93d-e5d1c5045fc8 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:13.481310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab32db9-a0ad-4242-b93d-e5d1c5045fc8 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:13.514687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a07963-9cfc-41f6-9d42-47880b39a00b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:13.518756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a07963-9cfc-41f6-9d42-47880b39a00b None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:13.518756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a07963-9cfc-41f6-9d42-47880b39a00b None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:13.518756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a07963-9cfc-41f6-9d42-47880b39a00b None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:13.518756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a07963-9cfc-41f6-9d42-47880b39a00b None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:13.518756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a07963-9cfc-41f6-9d42-47880b39a00b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:13.518756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a07963-9cfc-41f6-9d42-47880b39a00b None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:13.536033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 617/1240] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Jun 10 12:49:13 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:13.541794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff85fabf-ef72-49c1-b5dd-d04557dae316 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:13.643888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff85fabf-ef72-49c1-b5dd-d04557dae316 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:13.681328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b26588-757f-4962-8e4c-9be988ad1304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:13.681973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b26588-757f-4962-8e4c-9be988ad1304 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:13.682258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b26588-757f-4962-8e4c-9be988ad1304 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:13.682532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b26588-757f-4962-8e4c-9be988ad1304 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:13.741955 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:13.748731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b26588-757f-4962-8e4c-9be988ad1304 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:13.750217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b26588-757f-4962-8e4c-9be988ad1304 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:13.843528 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 624/1241] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:49:13 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jun 10 12:49:13.849413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a07963-9cfc-41f6-9d42-47880b39a00b None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:13.877066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:13.946541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a07963-9cfc-41f6-9d42-47880b39a00b None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:13.981378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca425d2-19d4-49cb-849c-0d2564777f38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:13.982682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca425d2-19d4-49cb-849c-0d2564777f38 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:13.983003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca425d2-19d4-49cb-849c-0d2564777f38 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:13.983294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca425d2-19d4-49cb-849c-0d2564777f38 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:13.994639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca425d2-19d4-49cb-849c-0d2564777f38 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:13.995042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca425d2-19d4-49cb-849c-0d2564777f38 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-9ca425d2-19d4-49cb-849c-0d2564777f38 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:14.012613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:14.013732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:14.014657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:14.014657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:14.014657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:14.014657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:14.014657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 618/1242] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:14.019801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b26588-757f-4962-8e4c-9be988ad1304 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:14.109678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:14.115223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b26588-757f-4962-8e4c-9be988ad1304 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:14.147470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faa43d9-177f-45b2-a159-601c35e3466a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:14.148150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faa43d9-177f-45b2-a159-601c35e3466a None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:14.148511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faa43d9-177f-45b2-a159-601c35e3466a None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:14.148691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0faa43d9-177f-45b2-a159-601c35e3466a None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:14.160300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0faa43d9-177f-45b2-a159-601c35e3466a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:14.160300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0faa43d9-177f-45b2-a159-601c35e3466a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:14.160300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0faa43d9-177f-45b2-a159-601c35e3466a None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:14.162658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 625/1243] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:14.168400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca425d2-19d4-49cb-849c-0d2564777f38 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:14.245251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:14.277525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca425d2-19d4-49cb-849c-0d2564777f38 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:14.314352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d109d49-00f2-417d-a586-caf12b59b544 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:14.315208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d109d49-00f2-417d-a586-caf12b59b544 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:14.315492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d109d49-00f2-417d-a586-caf12b59b544 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:14.315786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d109d49-00f2-417d-a586-caf12b59b544 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:14.322421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d109d49-00f2-417d-a586-caf12b59b544 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:14.323870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d109d49-00f2-417d-a586-caf12b59b544 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-5d109d49-00f2-417d-a586-caf12b59b544 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:49:14.337345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:14.338331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:14.339606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:49:14.339606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:49:14.339606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:49:14.339606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:49:14.339606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:49:14.339606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:49:14.339606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:14.339606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:14.339606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 619/1244] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:49:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:14.344784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faa43d9-177f-45b2-a159-601c35e3466a None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:14.442559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0faa43d9-177f-45b2-a159-601c35e3466a None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:14.476548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2475097e-5a75-4905-ba17-0642dd18436f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:14.477218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475097e-5a75-4905-ba17-0642dd18436f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:14.477499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475097e-5a75-4905-ba17-0642dd18436f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:14.477772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2475097e-5a75-4905-ba17-0642dd18436f None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:14.478537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2475097e-5a75-4905-ba17-0642dd18436f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:14.478838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2475097e-5a75-4905-ba17-0642dd18436f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:14.480375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2475097e-5a75-4905-ba17-0642dd18436f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:14.492127 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 626/1245] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:49:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:14.498171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d109d49-00f2-417d-a586-caf12b59b544 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:14.597213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d109d49-00f2-417d-a586-caf12b59b544 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:14.629349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c76b1fe-9ddd-4e7e-a458-a056f1e39018 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:14.629992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c76b1fe-9ddd-4e7e-a458-a056f1e39018 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:14.630274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c76b1fe-9ddd-4e7e-a458-a056f1e39018 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:14.630550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c76b1fe-9ddd-4e7e-a458-a056f1e39018 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:14.631372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c76b1fe-9ddd-4e7e-a458-a056f1e39018 None admin] RBAC: Adding query filter params (role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=0c315cfb1aa44f3c9a755f3cb8e5cd5d) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:14.631683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c76b1fe-9ddd-4e7e-a458-a056f1e39018 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:14.633163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c76b1fe-9ddd-4e7e-a458-a056f1e39018 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:14.639329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 620/1246] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:49:14 2022] GET /identity/v3/role_assignments?user.id=0c315cfb1aa44f3c9a755f3cb8e5cd5d&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=c7c592df9b6b4743a1ba6ca3b2a7fdbc => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:16.793216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f71f9-94d8-4984-8487-560c555e642a None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:16.793738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f71f9-94d8-4984-8487-560c555e642a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:16.793847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb5f71f9-94d8-4984-8487-560c555e642a None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:16.795691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 627/1247] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:16.800905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e939796d-a1a1-4cc2-b4de-ae74a1c99c21 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:16.801346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e939796d-a1a1-4cc2-b4de-ae74a1c99c21 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:16.801476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e939796d-a1a1-4cc2-b4de-ae74a1c99c21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:16.806468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:16.818847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e939796d-a1a1-4cc2-b4de-ae74a1c99c21 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:16.820037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e939796d-a1a1-4cc2-b4de-ae74a1c99c21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:16.884499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 621/1248] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:16 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:16.907994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5f71f9-94d8-4984-8487-560c555e642a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:16.926373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:17.005627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb5f71f9-94d8-4984-8487-560c555e642a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:17.038098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f21ad6f-3fe1-4480-8ccb-5c1d653d5666 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:17.038867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f21ad6f-3fe1-4480-8ccb-5c1d653d5666 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:17.039145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f21ad6f-3fe1-4480-8ccb-5c1d653d5666 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:17.039415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f21ad6f-3fe1-4480-8ccb-5c1d653d5666 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:17.049750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f21ad6f-3fe1-4480-8ccb-5c1d653d5666 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-9f21ad6f-3fe1-4480-8ccb-5c1d653d5666 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:49:17.057213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:17.058125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:17.059081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 628/1249] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:49:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:17.064460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e939796d-a1a1-4cc2-b4de-ae74a1c99c21 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:17.159056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e939796d-a1a1-4cc2-b4de-ae74a1c99c21 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:17.187016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:17.191587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27e22f4-4c89-4baa-a204-291aab692ad7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:17.192241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27e22f4-4c89-4baa-a204-291aab692ad7 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:17.192522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27e22f4-4c89-4baa-a204-291aab692ad7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:17.192818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27e22f4-4c89-4baa-a204-291aab692ad7 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:17.193552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27e22f4-4c89-4baa-a204-291aab692ad7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:17.193856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27e22f4-4c89-4baa-a204-291aab692ad7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:17.195236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27e22f4-4c89-4baa-a204-291aab692ad7 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:17.203012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 622/1250] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:49:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:17.208741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f21ad6f-3fe1-4480-8ccb-5c1d653d5666 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:17.284368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:17.306664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f21ad6f-3fe1-4480-8ccb-5c1d653d5666 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:17.338809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da42b9fc-f2f5-4e25-b8f8-f14650c04495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:17.339590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da42b9fc-f2f5-4e25-b8f8-f14650c04495 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:17.343188 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da42b9fc-f2f5-4e25-b8f8-f14650c04495 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:17.343188 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da42b9fc-f2f5-4e25-b8f8-f14650c04495 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:17.351226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da42b9fc-f2f5-4e25-b8f8-f14650c04495 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:17.352903 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da42b9fc-f2f5-4e25-b8f8-f14650c04495 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-da42b9fc-f2f5-4e25-b8f8-f14650c04495 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:17.370559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:17.376108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:17.377226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:17.377226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:17.377226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:17.377226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:17.377226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 629/1251] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:17.377675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27e22f4-4c89-4baa-a204-291aab692ad7 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:17.476201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27e22f4-4c89-4baa-a204-291aab692ad7 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:17.508446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a988c550-fad0-402c-a0e9-0c8c0a51e5d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:17.509134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a988c550-fad0-402c-a0e9-0c8c0a51e5d9 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:17.509419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a988c550-fad0-402c-a0e9-0c8c0a51e5d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:17.509697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a988c550-fad0-402c-a0e9-0c8c0a51e5d9 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:17.510410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a988c550-fad0-402c-a0e9-0c8c0a51e5d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:17.510718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a988c550-fad0-402c-a0e9-0c8c0a51e5d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:17.512092 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a988c550-fad0-402c-a0e9-0c8c0a51e5d9 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:17.524330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 623/1252] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:17.530609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da42b9fc-f2f5-4e25-b8f8-f14650c04495 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:17.629442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da42b9fc-f2f5-4e25-b8f8-f14650c04495 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:17.655522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:17.664675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56238945-d18b-429f-a606-254134f141be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:17.666093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56238945-d18b-429f-a606-254134f141be None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:17.666602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56238945-d18b-429f-a606-254134f141be None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:17.667106 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56238945-d18b-429f-a606-254134f141be None admin] PATH_INFO: `/v3/users/placement` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:17.675771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56238945-d18b-429f-a606-254134f141be None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:17.677254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56238945-d18b-429f-a606-254134f141be None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-56238945-d18b-429f-a606-254134f141be None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:49:17.690991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:17.692082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:17.693118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 630/1253] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:17 2022] GET /identity/v3/users/placement => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:17.698049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a988c550-fad0-402c-a0e9-0c8c0a51e5d9 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:17.753551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:17.796502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a988c550-fad0-402c-a0e9-0c8c0a51e5d9 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:17.830255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae40d46-495e-4031-bbb9-796302c552f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:17.830910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae40d46-495e-4031-bbb9-796302c552f3 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:17.831190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae40d46-495e-4031-bbb9-796302c552f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:17.831463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae40d46-495e-4031-bbb9-796302c552f3 None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:17.832284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae40d46-495e-4031-bbb9-796302c552f3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:17.832589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae40d46-495e-4031-bbb9-796302c552f3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:17.834069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae40d46-495e-4031-bbb9-796302c552f3 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:17.850383 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 624/1254] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Jun 10 12:49:17 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:17.856286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56238945-d18b-429f-a606-254134f141be None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:17.953308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56238945-d18b-429f-a606-254134f141be None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:17.985095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a163df1-0b66-4df1-a799-ee572af084be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:17.985726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a163df1-0b66-4df1-a799-ee572af084be None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:17.986009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a163df1-0b66-4df1-a799-ee572af084be None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:17.986282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a163df1-0b66-4df1-a799-ee572af084be None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:18.039439 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:18.050919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a163df1-0b66-4df1-a799-ee572af084be None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:18.052359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a163df1-0b66-4df1-a799-ee572af084be None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:18.146060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 631/1255] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:49:17 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:49:18.151815 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae40d46-495e-4031-bbb9-796302c552f3 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:18.246723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae40d46-495e-4031-bbb9-796302c552f3 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:18.259620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:18.279718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e22914c-cf2c-49f2-8211-0712270f15f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:18.280494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e22914c-cf2c-49f2-8211-0712270f15f0 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:18.280791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e22914c-cf2c-49f2-8211-0712270f15f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:18.281080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e22914c-cf2c-49f2-8211-0712270f15f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:18.290728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e22914c-cf2c-49f2-8211-0712270f15f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:18.292170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e22914c-cf2c-49f2-8211-0712270f15f0 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-2e22914c-cf2c-49f2-8211-0712270f15f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:18.309437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:18.311012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:18.312289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:18.312289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:18.312289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:18.312289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:18.312289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 625/1256] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:18.316845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a163df1-0b66-4df1-a799-ee572af084be None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:18.400932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:18.412120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a163df1-0b66-4df1-a799-ee572af084be None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:18.445212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05d2b06-ee0f-422d-8199-7de2d848cd62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:18.445858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05d2b06-ee0f-422d-8199-7de2d848cd62 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:18.446136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05d2b06-ee0f-422d-8199-7de2d848cd62 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:18.446449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c05d2b06-ee0f-422d-8199-7de2d848cd62 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:18.447156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05d2b06-ee0f-422d-8199-7de2d848cd62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:18.447461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05d2b06-ee0f-422d-8199-7de2d848cd62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:18.448840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c05d2b06-ee0f-422d-8199-7de2d848cd62 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:18.459632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 632/1257] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:18.465429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e22914c-cf2c-49f2-8211-0712270f15f0 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:18.563550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e22914c-cf2c-49f2-8211-0712270f15f0 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:18.596147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f8d20e-fb48-4841-acc0-5c20f6c1210f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:18.596988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f8d20e-fb48-4841-acc0-5c20f6c1210f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:18.597270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f8d20e-fb48-4841-acc0-5c20f6c1210f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:18.597544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f8d20e-fb48-4841-acc0-5c20f6c1210f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:18.603972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f8d20e-fb48-4841-acc0-5c20f6c1210f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:18.605526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f8d20e-fb48-4841-acc0-5c20f6c1210f None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-d7f8d20e-fb48-4841-acc0-5c20f6c1210f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:49:18.619572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:18.620601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:18.621697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:49:18.621697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:49:18.621697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:49:18.621697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:49:18.621697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:49:18.621697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:49:18.621697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:18.621697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:18.621697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 626/1258] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:49:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:18.626632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05d2b06-ee0f-422d-8199-7de2d848cd62 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:18.725907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c05d2b06-ee0f-422d-8199-7de2d848cd62 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:18.758504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44182c3e-fe8e-4e5a-bae1-1cd1d2899594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:18.759155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44182c3e-fe8e-4e5a-bae1-1cd1d2899594 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:18.759440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44182c3e-fe8e-4e5a-bae1-1cd1d2899594 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:18.759717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44182c3e-fe8e-4e5a-bae1-1cd1d2899594 None admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:18.760500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44182c3e-fe8e-4e5a-bae1-1cd1d2899594 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:18.760829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44182c3e-fe8e-4e5a-bae1-1cd1d2899594 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:18.764375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44182c3e-fe8e-4e5a-bae1-1cd1d2899594 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:18.774327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 633/1259] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:49:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:18.780922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f8d20e-fb48-4841-acc0-5c20f6c1210f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:18.806556 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:18.878537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f8d20e-fb48-4841-acc0-5c20f6c1210f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:18.912142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7325691d-230a-464a-b7b6-334297699afb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:18.912807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7325691d-230a-464a-b7b6-334297699afb None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:18.913095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7325691d-230a-464a-b7b6-334297699afb None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:18.913372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7325691d-230a-464a-b7b6-334297699afb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:18.914191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7325691d-230a-464a-b7b6-334297699afb None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=0c315cfb1aa44f3c9a755f3cb8e5cd5d) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:18.914499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7325691d-230a-464a-b7b6-334297699afb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:18.915927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7325691d-230a-464a-b7b6-334297699afb None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:18.921265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 627/1260] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:49:18 2022] GET /identity/v3/role_assignments?user.id=0c315cfb1aa44f3c9a755f3cb8e5cd5d&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=766799d1b6c049168590571f1a959b3b => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 12:49:21.035237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4571da-02f4-4c05-84dc-59c7fceb29d4 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:21.035818 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4571da-02f4-4c05-84dc-59c7fceb29d4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:21.035818 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4571da-02f4-4c05-84dc-59c7fceb29d4 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:21.037208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 634/1261] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:21.042548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b197760-ee05-4e38-8cfb-6a1d62ad6029 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:21.042985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b197760-ee05-4e38-8cfb-6a1d62ad6029 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:21.043087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b197760-ee05-4e38-8cfb-6a1d62ad6029 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:21.059306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5b197760-ee05-4e38-8cfb-6a1d62ad6029 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:21.061028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5b197760-ee05-4e38-8cfb-6a1d62ad6029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:21.126808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 628/1262] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:21 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:21.149544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4571da-02f4-4c05-84dc-59c7fceb29d4 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:21.160165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:21.247575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4571da-02f4-4c05-84dc-59c7fceb29d4 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:21.281345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2689de48-7691-405c-afd2-04bd558e923e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:21.282164 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2689de48-7691-405c-afd2-04bd558e923e None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:21.282449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2689de48-7691-405c-afd2-04bd558e923e None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:21.282731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2689de48-7691-405c-afd2-04bd558e923e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:21.292562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2689de48-7691-405c-afd2-04bd558e923e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-2689de48-7691-405c-afd2-04bd558e923e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:49:21.300602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:21.303706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:21.304813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 635/1263] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:49:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:21.307835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b197760-ee05-4e38-8cfb-6a1d62ad6029 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:21.327510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:21.405070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b197760-ee05-4e38-8cfb-6a1d62ad6029 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:21.438867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43b031-1f5c-40f7-884c-a0b5a2d60e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:21.439569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43b031-1f5c-40f7-884c-a0b5a2d60e03 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:21.439859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43b031-1f5c-40f7-884c-a0b5a2d60e03 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:21.440138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed43b031-1f5c-40f7-884c-a0b5a2d60e03 None admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:21.440909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed43b031-1f5c-40f7-884c-a0b5a2d60e03 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:21.441217 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed43b031-1f5c-40f7-884c-a0b5a2d60e03 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:21.442587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed43b031-1f5c-40f7-884c-a0b5a2d60e03 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:21.450421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 629/1264] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:49:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:21.456152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2689de48-7691-405c-afd2-04bd558e923e None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:21.520574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:21.553379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2689de48-7691-405c-afd2-04bd558e923e None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:21.584404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558b2aa5-7ee2-43e5-a836-9d830df0608b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:21.585198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558b2aa5-7ee2-43e5-a836-9d830df0608b None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:21.585481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558b2aa5-7ee2-43e5-a836-9d830df0608b None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:21.585756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558b2aa5-7ee2-43e5-a836-9d830df0608b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:21.598527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558b2aa5-7ee2-43e5-a836-9d830df0608b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:21.600017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558b2aa5-7ee2-43e5-a836-9d830df0608b None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-558b2aa5-7ee2-43e5-a836-9d830df0608b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:21.617123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:21.618161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:21.619110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:21.619110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:21.619110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:21.619110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:21.619110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 636/1265] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:21.623934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43b031-1f5c-40f7-884c-a0b5a2d60e03 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:21.707857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:21.726890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed43b031-1f5c-40f7-884c-a0b5a2d60e03 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:21.760181 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8b0bbc-898f-4457-946d-a39b70e797b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:21.760866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8b0bbc-898f-4457-946d-a39b70e797b7 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:21.761153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8b0bbc-898f-4457-946d-a39b70e797b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:21.761431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8b0bbc-898f-4457-946d-a39b70e797b7 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:21.762151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8b0bbc-898f-4457-946d-a39b70e797b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:21.762460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8b0bbc-898f-4457-946d-a39b70e797b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:21.763863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8b0bbc-898f-4457-946d-a39b70e797b7 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:21.775780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 630/1266] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:21.782213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558b2aa5-7ee2-43e5-a836-9d830df0608b None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:21.878944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558b2aa5-7ee2-43e5-a836-9d830df0608b None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:21.898264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:21.913100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76efa3ec-115b-4bce-a673-f0672f017366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:21.913884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76efa3ec-115b-4bce-a673-f0672f017366 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:21.914162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76efa3ec-115b-4bce-a673-f0672f017366 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:21.914436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76efa3ec-115b-4bce-a673-f0672f017366 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:21.921880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76efa3ec-115b-4bce-a673-f0672f017366 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:21.923314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76efa3ec-115b-4bce-a673-f0672f017366 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-76efa3ec-115b-4bce-a673-f0672f017366 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:49:21.937148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:21.938237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:21.939391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 637/1267] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:21 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:21.944200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8b0bbc-898f-4457-946d-a39b70e797b7 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:22.042298 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8b0bbc-898f-4457-946d-a39b70e797b7 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:22.075287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d758a-33bc-47d3-9abe-e2638955f12c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:22.075944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d758a-33bc-47d3-9abe-e2638955f12c None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:22.076229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d758a-33bc-47d3-9abe-e2638955f12c None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:22.076501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362d758a-33bc-47d3-9abe-e2638955f12c None admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:22.077355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362d758a-33bc-47d3-9abe-e2638955f12c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:22.077666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362d758a-33bc-47d3-9abe-e2638955f12c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:22.079091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362d758a-33bc-47d3-9abe-e2638955f12c None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:22.087344 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:22.095366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 631/1268] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Jun 10 12:49:21 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:22.101007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76efa3ec-115b-4bce-a673-f0672f017366 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:22.197032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76efa3ec-115b-4bce-a673-f0672f017366 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:22.228898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae87f7b-c777-49b0-b172-b80d83301a90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:22.229668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae87f7b-c777-49b0-b172-b80d83301a90 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:22.229947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae87f7b-c777-49b0-b172-b80d83301a90 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:22.230220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae87f7b-c777-49b0-b172-b80d83301a90 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:22.240743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae87f7b-c777-49b0-b172-b80d83301a90 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:22.242239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eae87f7b-c777-49b0-b172-b80d83301a90 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-eae87f7b-c777-49b0-b172-b80d83301a90 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:22.259419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:22.260314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:22.261316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:22.261316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:22.261316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:22.261316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:22.261316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 638/1269] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:22.266582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d758a-33bc-47d3-9abe-e2638955f12c None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:22.369595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362d758a-33bc-47d3-9abe-e2638955f12c None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:22.402887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5be0df-94a2-41d3-b04c-d589e189f08e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:22.404079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5be0df-94a2-41d3-b04c-d589e189f08e None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:22.404352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5be0df-94a2-41d3-b04c-d589e189f08e None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:22.404619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f5be0df-94a2-41d3-b04c-d589e189f08e None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:22.405352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5be0df-94a2-41d3-b04c-d589e189f08e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:22.405654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5be0df-94a2-41d3-b04c-d589e189f08e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:22.407019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f5be0df-94a2-41d3-b04c-d589e189f08e None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:22.417965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 632/1270] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:22.424201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae87f7b-c777-49b0-b172-b80d83301a90 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:22.454356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:22.519668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae87f7b-c777-49b0-b172-b80d83301a90 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:22.556330 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ae8ee-b7e0-45e8-b042-04debd97c7d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:22.557159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ae8ee-b7e0-45e8-b042-04debd97c7d2 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:22.557440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ae8ee-b7e0-45e8-b042-04debd97c7d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:22.557715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08ae8ee-b7e0-45e8-b042-04debd97c7d2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:22.564410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08ae8ee-b7e0-45e8-b042-04debd97c7d2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:22.565885 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a08ae8ee-b7e0-45e8-b042-04debd97c7d2 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-a08ae8ee-b7e0-45e8-b042-04debd97c7d2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:49:22.579205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:22.581774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:22.582755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:49:22.582755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:49:22.582755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:49:22.582755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:49:22.582755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:49:22.582755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:49:22.582755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:22.582755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:22.582755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 639/1271] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:49:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:49:22.586174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5be0df-94a2-41d3-b04c-d589e189f08e None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:22.644621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:22.683989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f5be0df-94a2-41d3-b04c-d589e189f08e None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:22.722090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bba097-034e-49bd-9ced-32794a06668d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:22.722675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bba097-034e-49bd-9ced-32794a06668d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:22.722962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bba097-034e-49bd-9ced-32794a06668d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:22.723242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bba097-034e-49bd-9ced-32794a06668d None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:22.724031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bba097-034e-49bd-9ced-32794a06668d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:22.724342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bba097-034e-49bd-9ced-32794a06668d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:22.725875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bba097-034e-49bd-9ced-32794a06668d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:22.737870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 633/1272] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:49:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:22.743789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ae8ee-b7e0-45e8-b042-04debd97c7d2 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:22.834417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:22.843163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a08ae8ee-b7e0-45e8-b042-04debd97c7d2 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:22.875700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66b8ec0-ca22-43ab-aca2-e966a027d975 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:22.876570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b8ec0-ca22-43ab-aca2-e966a027d975 None admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:22.876874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b8ec0-ca22-43ab-aca2-e966a027d975 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:22.877146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b8ec0-ca22-43ab-aca2-e966a027d975 None admin] PATH_INFO: `/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/0c315cfb1aa44f3c9a755f3cb8e5cd5d/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:22.879043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66b8ec0-ca22-43ab-aca2-e966a027d975 None admin] RBAC: Authorizing `identity:create_grant(project_id=c1103ec5f93b4ee1a5d72cff4c40178a, user_id=0c315cfb1aa44f3c9a755f3cb8e5cd5d, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:22.880473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66b8ec0-ca22-43ab-aca2-e966a027d975 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:22.959821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 640/1273] 10.222.0.25 () {58 vars in 1439 bytes} [Fri Jun 10 12:49:22 2022] PUT /identity/v3/projects/c1103ec5f93b4ee1a5d72cff4c40178a/users/0c315cfb1aa44f3c9a755f3cb8e5cd5d/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 12:49:25.071487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f8de7d-cb7d-4451-84fa-59bd32960329 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:25.072014 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f8de7d-cb7d-4451-84fa-59bd32960329 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:25.072106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f8de7d-cb7d-4451-84fa-59bd32960329 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:25.073507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 634/1274] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:25.078629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b873cad-b946-4220-ac25-7fe4fbb28d04 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:25.079142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b873cad-b946-4220-ac25-7fe4fbb28d04 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:25.079231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b873cad-b946-4220-ac25-7fe4fbb28d04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:25.095410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6b873cad-b946-4220-ac25-7fe4fbb28d04 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:25.096597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6b873cad-b946-4220-ac25-7fe4fbb28d04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:25.160449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 641/1275] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:25 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:25.183825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f8de7d-cb7d-4451-84fa-59bd32960329 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:25.283753 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f8de7d-cb7d-4451-84fa-59bd32960329 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:25.301544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:25.317564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7a1b6-e24c-491f-be73-18123e93cd4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:25.318346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7a1b6-e24c-491f-be73-18123e93cd4f None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:25.318661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7a1b6-e24c-491f-be73-18123e93cd4f None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:25.318942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f7a1b6-e24c-491f-be73-18123e93cd4f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:25.329079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f7a1b6-e24c-491f-be73-18123e93cd4f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-d1f7a1b6-e24c-491f-be73-18123e93cd4f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(role_id) Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jun 10 12:49:25.340118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:25.341847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:25.342874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 635/1276] 10.222.0.25 () {56 vars in 1096 bytes} [Fri Jun 10 12:49:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:25.344412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b873cad-b946-4220-ac25-7fe4fbb28d04 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:25.423012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:25.440554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b873cad-b946-4220-ac25-7fe4fbb28d04 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:25.472622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b142fb6d-fb90-48dd-ad39-bbafe7af39af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:25.473308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b142fb6d-fb90-48dd-ad39-bbafe7af39af None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:25.473595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b142fb6d-fb90-48dd-ad39-bbafe7af39af None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:25.473876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b142fb6d-fb90-48dd-ad39-bbafe7af39af None admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:25.474618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b142fb6d-fb90-48dd-ad39-bbafe7af39af None admin] RBAC: Adding query filter params (name=admin) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:25.474929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b142fb6d-fb90-48dd-ad39-bbafe7af39af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:25.476314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b142fb6d-fb90-48dd-ad39-bbafe7af39af None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:25.484034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 642/1277] 10.222.0.25 () {56 vars in 1099 bytes} [Fri Jun 10 12:49:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:25.489731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7a1b6-e24c-491f-be73-18123e93cd4f None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:25.587383 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f7a1b6-e24c-491f-be73-18123e93cd4f None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:25.619755 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a789f3a-c30e-4269-b885-d4108ad347ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:25.620529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a789f3a-c30e-4269-b885-d4108ad347ed None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:25.620839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a789f3a-c30e-4269-b885-d4108ad347ed None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:25.621116 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a789f3a-c30e-4269-b885-d4108ad347ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:25.631926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a789f3a-c30e-4269-b885-d4108ad347ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:25.633400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a789f3a-c30e-4269-b885-d4108ad347ed None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-7a789f3a-c30e-4269-b885-d4108ad347ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:25.650143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:25.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:25.652461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:25.652461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:25.652461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:25.652461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:25.652461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 636/1278] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jun 10 12:49:25.657089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b142fb6d-fb90-48dd-ad39-bbafe7af39af None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:25.758261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b142fb6d-fb90-48dd-ad39-bbafe7af39af None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:25.789761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c6307-7093-42d9-b842-1b4d26ffc536 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:25.790403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c6307-7093-42d9-b842-1b4d26ffc536 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:25.790682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c6307-7093-42d9-b842-1b4d26ffc536 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:25.790954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c6307-7093-42d9-b842-1b4d26ffc536 None admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:25.791655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c6307-7093-42d9-b842-1b4d26ffc536 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:25.791956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c6307-7093-42d9-b842-1b4d26ffc536 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:25.793369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c6307-7093-42d9-b842-1b4d26ffc536 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:25.805076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 643/1279] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:25.811400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a789f3a-c30e-4269-b885-d4108ad347ed None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:25.842648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:25.909688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a789f3a-c30e-4269-b885-d4108ad347ed None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:25.944744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045b8d9e-6889-4b35-8e07-626cf387b1bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:25.945563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045b8d9e-6889-4b35-8e07-626cf387b1bf None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:25.945847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045b8d9e-6889-4b35-8e07-626cf387b1bf None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:25.946126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045b8d9e-6889-4b35-8e07-626cf387b1bf None admin] PATH_INFO: `/v3/users/placement` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:25.954382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045b8d9e-6889-4b35-8e07-626cf387b1bf None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:25.955837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045b8d9e-6889-4b35-8e07-626cf387b1bf None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-045b8d9e-6889-4b35-8e07-626cf387b1bf None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_user(user_id) Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jun 10 12:49:25.969945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:25.971681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:25.972853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 637/1280] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:25 2022] GET /identity/v3/users/placement => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:25.976951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c6307-7093-42d9-b842-1b4d26ffc536 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:25.980585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:26.071817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c6307-7093-42d9-b842-1b4d26ffc536 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:26.104986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd94bf7-20d9-4111-b1fa-0f3856f80a59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:26.105613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd94bf7-20d9-4111-b1fa-0f3856f80a59 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:26.105844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd94bf7-20d9-4111-b1fa-0f3856f80a59 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:26.106682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd94bf7-20d9-4111-b1fa-0f3856f80a59 None admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:26.107515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd94bf7-20d9-4111-b1fa-0f3856f80a59 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:26.107860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd94bf7-20d9-4111-b1fa-0f3856f80a59 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:26.109313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd94bf7-20d9-4111-b1fa-0f3856f80a59 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:26.126330 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 644/1281] 10.222.0.25 () {56 vars in 1143 bytes} [Fri Jun 10 12:49:25 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:26.132191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045b8d9e-6889-4b35-8e07-626cf387b1bf None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:26.220317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:26.229217 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045b8d9e-6889-4b35-8e07-626cf387b1bf None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:26.261148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8fa5dd-745c-4564-9504-c31bf52f891a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:26.261775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8fa5dd-745c-4564-9504-c31bf52f891a None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:26.262080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8fa5dd-745c-4564-9504-c31bf52f891a None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:26.262357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8fa5dd-745c-4564-9504-c31bf52f891a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:26.329394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8fa5dd-745c-4564-9504-c31bf52f891a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:26.330865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df8fa5dd-745c-4564-9504-c31bf52f891a None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:26.419704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:26.429744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 638/1282] 10.222.0.25 () {58 vars in 1282 bytes} [Fri Jun 10 12:49:26 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jun 10 12:49:26.435529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd94bf7-20d9-4111-b1fa-0f3856f80a59 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:26.494346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:26.532139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd94bf7-20d9-4111-b1fa-0f3856f80a59 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:26.564617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74ecaf1-7f49-4433-b4c7-cd03a5a0ffef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:26.565441 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74ecaf1-7f49-4433-b4c7-cd03a5a0ffef None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:26.565721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74ecaf1-7f49-4433-b4c7-cd03a5a0ffef None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:26.565994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c74ecaf1-7f49-4433-b4c7-cd03a5a0ffef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:26.576984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74ecaf1-7f49-4433-b4c7-cd03a5a0ffef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:26.578427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c74ecaf1-7f49-4433-b4c7-cd03a5a0ffef None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-c74ecaf1-7f49-4433-b4c7-cd03a5a0ffef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jun 10 12:49:26.595731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:26.596867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:26.598068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jun 10 12:49:26.598068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jun 10 12:49:26.598068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jun 10 12:49:26.598068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:26.602022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 645/1283] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:49:26 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:26.603315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8fa5dd-745c-4564-9504-c31bf52f891a None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:26.703778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df8fa5dd-745c-4564-9504-c31bf52f891a None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:26.736868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8302d1f8-b5aa-46ee-8eae-28097b9a4261 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:26.737516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8302d1f8-b5aa-46ee-8eae-28097b9a4261 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:26.737796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8302d1f8-b5aa-46ee-8eae-28097b9a4261 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:26.738068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8302d1f8-b5aa-46ee-8eae-28097b9a4261 None admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:26.738769 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8302d1f8-b5aa-46ee-8eae-28097b9a4261 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:26.739069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8302d1f8-b5aa-46ee-8eae-28097b9a4261 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:26.740427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8302d1f8-b5aa-46ee-8eae-28097b9a4261 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:26.751845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 639/1284] 10.222.0.25 () {56 vars in 1109 bytes} [Fri Jun 10 12:49:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:26.757522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74ecaf1-7f49-4433-b4c7-cd03a5a0ffef None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:26.856842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c74ecaf1-7f49-4433-b4c7-cd03a5a0ffef None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:26.869702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:26.889458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b72ee-c378-4015-bc13-e6e41116c363 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:26.890479 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b72ee-c378-4015-bc13-e6e41116c363 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:26.890770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b72ee-c378-4015-bc13-e6e41116c363 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:26.891041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b43b72ee-c378-4015-bc13-e6e41116c363 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:26.897444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b72ee-c378-4015-bc13-e6e41116c363 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:26.898889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b43b72ee-c378-4015-bc13-e6e41116c363 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-b43b72ee-c378-4015-bc13-e6e41116c363 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(project_id) Jun 10 12:49:26.912883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:26.919866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:26.921067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 12:49:26.921067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 12:49:26.921067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 12:49:26.921067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 12:49:26.921067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 12:49:26.921067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 12:49:26.921067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jun 10 12:49:26.921067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 12:49:26.921067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 646/1285] 10.222.0.25 () {56 vars in 1111 bytes} [Fri Jun 10 12:49:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:26.921695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8302d1f8-b5aa-46ee-8eae-28097b9a4261 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:26.981358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:27.031123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8302d1f8-b5aa-46ee-8eae-28097b9a4261 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:27.063258 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b725249-17f5-4be9-a0de-89921c2390c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:27.063907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b725249-17f5-4be9-a0de-89921c2390c2 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:27.064188 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b725249-17f5-4be9-a0de-89921c2390c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:27.064464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b725249-17f5-4be9-a0de-89921c2390c2 None admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:27.065269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b725249-17f5-4be9-a0de-89921c2390c2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:27.065580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b725249-17f5-4be9-a0de-89921c2390c2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:27.067006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b725249-17f5-4be9-a0de-89921c2390c2 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:27.079082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 640/1286] 10.222.0.25 () {56 vars in 1148 bytes} [Fri Jun 10 12:49:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:27.085055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b72ee-c378-4015-bc13-e6e41116c363 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:27.182763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b43b72ee-c378-4015-bc13-e6e41116c363 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:27.214944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c68c47-d991-4687-936c-158d328c7a5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:27.215579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c68c47-d991-4687-936c-158d328c7a5f None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:27.215860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c68c47-d991-4687-936c-158d328c7a5f None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:27.216134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c68c47-d991-4687-936c-158d328c7a5f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:27.216983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c68c47-d991-4687-936c-158d328c7a5f None admin] RBAC: Adding query filter params (role.id=766799d1b6c049168590571f1a959b3b, scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a, user.id=0c315cfb1aa44f3c9a755f3cb8e5cd5d) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:27.217292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c68c47-d991-4687-936c-158d328c7a5f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:27.218714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c68c47-d991-4687-936c-158d328c7a5f None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:27.224367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 647/1287] 10.222.0.25 () {56 vars in 1374 bytes} [Fri Jun 10 12:49:27 2022] GET /identity/v3/role_assignments?user.id=0c315cfb1aa44f3c9a755f3cb8e5cd5d&scope.project.id=c1103ec5f93b4ee1a5d72cff4c40178a&role.id=766799d1b6c049168590571f1a959b3b => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:29.453236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a815c9-01f7-40cc-9f79-931e78a2b7c3 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:29.453784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a815c9-01f7-40cc-9f79-931e78a2b7c3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:29.453784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a815c9-01f7-40cc-9f79-931e78a2b7c3 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:29.455152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 641/1288] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:29.460299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfff942-179f-4922-8332-2b90825e3d35 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:29.460669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfff942-179f-4922-8332-2b90825e3d35 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:29.460888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdfff942-179f-4922-8332-2b90825e3d35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:29.476978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-fdfff942-179f-4922-8332-2b90825e3d35 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:29.478152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-fdfff942-179f-4922-8332-2b90825e3d35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:29.507333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:29.542143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 648/1289] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:29 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:29.564954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a815c9-01f7-40cc-9f79-931e78a2b7c3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:29.664246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a815c9-01f7-40cc-9f79-931e78a2b7c3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:29.679016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:29.707764 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab5ec8-1334-4173-9c6e-2e6caab50ad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:29.708582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab5ec8-1334-4173-9c6e-2e6caab50ad2 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:29.708889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab5ec8-1334-4173-9c6e-2e6caab50ad2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:29.709164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67ab5ec8-1334-4173-9c6e-2e6caab50ad2 None admin] PATH_INFO: `/v3/services/placement` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:29.709814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab5ec8-1334-4173-9c6e-2e6caab50ad2 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:29.711183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67ab5ec8-1334-4173-9c6e-2e6caab50ad2 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-67ab5ec8-1334-4173-9c6e-2e6caab50ad2 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:29.722701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:29.723775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:29.724719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 642/1290] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:49:29 2022] GET /identity/v3/services/placement => generated 90 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:29.733258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfff942-179f-4922-8332-2b90825e3d35 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:29.824253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdfff942-179f-4922-8332-2b90825e3d35 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:29.856833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66327936-4d4e-46a8-913a-3b330d6c88cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:29.857483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66327936-4d4e-46a8-913a-3b330d6c88cc None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:29.857754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66327936-4d4e-46a8-913a-3b330d6c88cc None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:29.858028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66327936-4d4e-46a8-913a-3b330d6c88cc None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:29.858719 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66327936-4d4e-46a8-913a-3b330d6c88cc None admin] RBAC: Adding query filter params (name=placement) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:29.859023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66327936-4d4e-46a8-913a-3b330d6c88cc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:29.860378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66327936-4d4e-46a8-913a-3b330d6c88cc None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:29.869366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 649/1291] 10.222.0.25 () {56 vars in 1116 bytes} [Fri Jun 10 12:49:29 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:29.870982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab5ec8-1334-4173-9c6e-2e6caab50ad2 None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:29.971651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67ab5ec8-1334-4173-9c6e-2e6caab50ad2 None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:30.004748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcd9831-a885-4842-84e4-57608574e8d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:30.005437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcd9831-a885-4842-84e4-57608574e8d9 None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:30.005718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcd9831-a885-4842-84e4-57608574e8d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:30.006138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcd9831-a885-4842-84e4-57608574e8d9 None admin] PATH_INFO: `/v3/services` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:30.006974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcd9831-a885-4842-84e4-57608574e8d9 None admin] RBAC: Adding query filter params (type=placement) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:30.007287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcd9831-a885-4842-84e4-57608574e8d9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:30.008676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcd9831-a885-4842-84e4-57608574e8d9 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:30.015887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 643/1292] 10.222.0.25 () {56 vars in 1116 bytes} [Fri Jun 10 12:49:29 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:32.088269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178d7273-b87b-4aaf-bbf2-d5d1029ea05d None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:32.088827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178d7273-b87b-4aaf-bbf2-d5d1029ea05d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:32.088924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178d7273-b87b-4aaf-bbf2-d5d1029ea05d None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:32.090748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 650/1293] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:32.095890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62037768-ee81-4661-a617-0f2a806e42f9 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:32.096304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62037768-ee81-4661-a617-0f2a806e42f9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:32.096459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62037768-ee81-4661-a617-0f2a806e42f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:32.113572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-62037768-ee81-4661-a617-0f2a806e42f9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:32.115695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-62037768-ee81-4661-a617-0f2a806e42f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:32.142996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:32.178704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 644/1294] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:32 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:32.200905 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178d7273-b87b-4aaf-bbf2-d5d1029ea05d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:32.232138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:32.300143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178d7273-b87b-4aaf-bbf2-d5d1029ea05d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:32.333446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8d14d5-5512-465f-b257-0fa2f65ea441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:32.334101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8d14d5-5512-465f-b257-0fa2f65ea441 None admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:32.334891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8d14d5-5512-465f-b257-0fa2f65ea441 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:32.334891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8d14d5-5512-465f-b257-0fa2f65ea441 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:32.335396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8d14d5-5512-465f-b257-0fa2f65ea441 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:32.336841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8d14d5-5512-465f-b257-0fa2f65ea441 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:32.463128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 651/1295] 10.222.0.25 () {60 vars in 1141 bytes} [Fri Jun 10 12:49:32 2022] POST /identity/v3/services => generated 246 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 12:49:34.582605 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8941e72a-7f32-4d81-879a-7e3ad99fc1ca None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:34.583240 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8941e72a-7f32-4d81-879a-7e3ad99fc1ca None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:34.583240 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8941e72a-7f32-4d81-879a-7e3ad99fc1ca None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:34.584610 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 645/1296] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:34.590520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d42e25b-6a8f-426d-ab0e-1187387e58db None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:34.591062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d42e25b-6a8f-426d-ab0e-1187387e58db None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:34.591062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d42e25b-6a8f-426d-ab0e-1187387e58db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:34.608894 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3d42e25b-6a8f-426d-ab0e-1187387e58db None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:34.610113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3d42e25b-6a8f-426d-ab0e-1187387e58db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:34.687021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 652/1297] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:34 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jun 10 12:49:34.710141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8941e72a-7f32-4d81-879a-7e3ad99fc1ca None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:34.814954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8941e72a-7f32-4d81-879a-7e3ad99fc1ca None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:34.847385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb16d9e-3583-485e-ade4-28c2df3657cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:34.848112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb16d9e-3583-485e-ade4-28c2df3657cd None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:34.848389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb16d9e-3583-485e-ade4-28c2df3657cd None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:34.848662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb16d9e-3583-485e-ade4-28c2df3657cd None admin] PATH_INFO: `/v3/services/placement` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:34.849333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb16d9e-3583-485e-ade4-28c2df3657cd None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:34.850741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb16d9e-3583-485e-ade4-28c2df3657cd None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-4eb16d9e-3583-485e-ade4-28c2df3657cd None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:34.863119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:34.864346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:34.865318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 646/1298] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:49:34 2022] GET /identity/v3/services/placement => generated 90 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:34.870245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d42e25b-6a8f-426d-ab0e-1187387e58db None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:34.889051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:34.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d42e25b-6a8f-426d-ab0e-1187387e58db None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:34.997567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11249134-4d26-4906-a754-9acecce32b42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:34.998214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11249134-4d26-4906-a754-9acecce32b42 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:34.998494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11249134-4d26-4906-a754-9acecce32b42 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:34.998884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11249134-4d26-4906-a754-9acecce32b42 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:34.999583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11249134-4d26-4906-a754-9acecce32b42 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:34.999888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11249134-4d26-4906-a754-9acecce32b42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:35.001282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11249134-4d26-4906-a754-9acecce32b42 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:35.006334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 653/1299] 10.222.0.25 () {56 vars in 1116 bytes} [Fri Jun 10 12:49:34 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:35.012362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb16d9e-3583-485e-ade4-28c2df3657cd None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:35.044563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:35.110341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb16d9e-3583-485e-ade4-28c2df3657cd None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:35.144254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b163d631-54d0-4153-ab8d-c75bd4fe178d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:35.144920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b163d631-54d0-4153-ab8d-c75bd4fe178d None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:35.145206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b163d631-54d0-4153-ab8d-c75bd4fe178d None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:35.145486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b163d631-54d0-4153-ab8d-c75bd4fe178d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:35.146263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b163d631-54d0-4153-ab8d-c75bd4fe178d None admin] RBAC: Adding query filter params (interface=public, service_id=5a0e76fe319a4b0981b5601a3fc8691f, region_id=RegionOne) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:35.146576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b163d631-54d0-4153-ab8d-c75bd4fe178d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:35.147940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b163d631-54d0-4153-ab8d-c75bd4fe178d None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:35.153061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 647/1300] 10.222.0.25 () {56 vars in 1251 bytes} [Fri Jun 10 12:49:35 2022] GET /identity/v3/endpoints?service_id=5a0e76fe319a4b0981b5601a3fc8691f&interface=public®ion_id=RegionOne => generated 194 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:35.158759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11249134-4d26-4906-a754-9acecce32b42 None admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:35.237030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:35.254907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11249134-4d26-4906-a754-9acecce32b42 None admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:35.287365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96842385-3c38-4ed2-9ee1-50d7343c77f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:35.288015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96842385-3c38-4ed2-9ee1-50d7343c77f1 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:35.288300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96842385-3c38-4ed2-9ee1-50d7343c77f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:35.288578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96842385-3c38-4ed2-9ee1-50d7343c77f1 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:35.289287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96842385-3c38-4ed2-9ee1-50d7343c77f1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:35.290653 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96842385-3c38-4ed2-9ee1-50d7343c77f1 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:35.297236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 654/1301] 10.222.0.25 () {56 vars in 1087 bytes} [Fri Jun 10 12:49:35 2022] GET /identity/v3/services => generated 2191 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 12:49:37.406323 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659124a-bfa7-440a-8b38-843d68c9806a None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:37.406850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659124a-bfa7-440a-8b38-843d68c9806a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:37.406949 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659124a-bfa7-440a-8b38-843d68c9806a None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:37.408275 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 648/1302] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:49:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:49:37.413614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5b9f71-2762-459d-a6ae-52bab9489c4d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:37.414051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5b9f71-2762-459d-a6ae-52bab9489c4d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:37.414175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5b9f71-2762-459d-a6ae-52bab9489c4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:37.430434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-cc5b9f71-2762-459d-a6ae-52bab9489c4d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:49:37.431609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-cc5b9f71-2762-459d-a6ae-52bab9489c4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:49:37.494788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 655/1303] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:49:37 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jun 10 12:49:37.518993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659124a-bfa7-440a-8b38-843d68c9806a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:37.611830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:37.618254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659124a-bfa7-440a-8b38-843d68c9806a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:37.651082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f992e488-5d93-4373-bcaf-f0e4586c19bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:37.651871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f992e488-5d93-4373-bcaf-f0e4586c19bd None admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:37.652151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f992e488-5d93-4373-bcaf-f0e4586c19bd None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:37.652425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f992e488-5d93-4373-bcaf-f0e4586c19bd None admin] PATH_INFO: `/v3/services/placement` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:37.653101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f992e488-5d93-4373-bcaf-f0e4586c19bd None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:37.654464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f992e488-5d93-4373-bcaf-f0e4586c19bd None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application [None req-f992e488-5d93-4373-bcaf-f0e4586c19bd None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 12:49:37.666059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._get_service(service_id) Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 12:49:37.667228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application with Lock( Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self._enter() Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application return self.creator() Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application created_value = creator( Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: ERROR keystone.server.flask.application Jun 10 12:49:37.668570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 649/1304] 10.222.0.25 () {56 vars in 1117 bytes} [Fri Jun 10 12:49:37 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jun 10 12:49:37.673679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5b9f71-2762-459d-a6ae-52bab9489c4d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:37.746470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:37.767865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5b9f71-2762-459d-a6ae-52bab9489c4d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:37.800384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fcb532-4ff2-4c96-b55c-28c968750000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:37.801085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fcb532-4ff2-4c96-b55c-28c968750000 None admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:37.801366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fcb532-4ff2-4c96-b55c-28c968750000 None admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:37.801645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fcb532-4ff2-4c96-b55c-28c968750000 None admin] PATH_INFO: `/v3/services` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:37.802354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fcb532-4ff2-4c96-b55c-28c968750000 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 12:49:37.802662 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fcb532-4ff2-4c96-b55c-28c968750000 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:37.804045 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6fcb532-4ff2-4c96-b55c-28c968750000 None admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:37.810349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 656/1305] 10.222.0.25 () {56 vars in 1116 bytes} [Fri Jun 10 12:49:37 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 12:49:37.818352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f992e488-5d93-4373-bcaf-f0e4586c19bd None admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:49:37.915885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f992e488-5d93-4373-bcaf-f0e4586c19bd None admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:49:37.949068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa63f1d-45f5-43d6-bb31-5bcaa50ff7b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:49:37.949711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa63f1d-45f5-43d6-bb31-5bcaa50ff7b2 None admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:49:37.949994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa63f1d-45f5-43d6-bb31-5bcaa50ff7b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:49:37.950273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa63f1d-45f5-43d6-bb31-5bcaa50ff7b2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:49:37.951013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa63f1d-45f5-43d6-bb31-5bcaa50ff7b2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:49:37.952389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa63f1d-45f5-43d6-bb31-5bcaa50ff7b2 None admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:49:38.039719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 650/1306] 10.222.0.25 () {60 vars in 1144 bytes} [Fri Jun 10 12:49:37 2022] POST /identity/v3/endpoints => generated 331 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 12:49:47.806868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=30 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:49:48.035118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:56:49.564878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1a7b8d-c140-4cea-aeed-874c17158db7 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:49.565960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1a7b8d-c140-4cea-aeed-874c17158db7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:49.566468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1a7b8d-c140-4cea-aeed-874c17158db7 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:49.573620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbc6c5-c468-46d4-8775-440d258602e9 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:49.574443 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbc6c5-c468-46d4-8775-440d258602e9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:49.574951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eddbc6c5-c468-46d4-8775-440d258602e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:49.578521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 657/1307] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:56:49 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:56:49.707676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-eddbc6c5-c468-46d4-8775-440d258602e9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:56:49.709808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-eddbc6c5-c468-46d4-8775-440d258602e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:49.908387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 651/1308] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:56:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 336 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 12:56:49.961683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4520c1e-4fb2-41d3-848a-84d9af5b856c None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:49.962334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4520c1e-4fb2-41d3-848a-84d9af5b856c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:49.962561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4520c1e-4fb2-41d3-848a-84d9af5b856c None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:49.964343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 658/1309] 10.222.0.25 () {54 vars in 964 bytes} [Fri Jun 10 12:56:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:56:49.970982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890faa87-6d28-4804-a34a-76cc194cd933 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:49.971284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890faa87-6d28-4804-a34a-76cc194cd933 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:49.971549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-890faa87-6d28-4804-a34a-76cc194cd933 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:50.051057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-890faa87-6d28-4804-a34a-76cc194cd933 None None] MFA Rules not processed for user `d6794257c90e40bca2c8efd31b0c791a`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:56:50.052373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-890faa87-6d28-4804-a34a-76cc194cd933 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:50.074504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:56:50.133802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 652/1310] 10.222.0.25 () {58 vars in 1061 bytes} [Fri Jun 10 12:56:49 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:56:50.143364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4520c1e-4fb2-41d3-848a-84d9af5b856c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:56:50.145373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d4520c1e-4fb2-41d3-848a-84d9af5b856c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:50.264579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4520c1e-4fb2-41d3-848a-84d9af5b856c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:56:50.347790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52399559-5a97-4e96-b5ca-7b058c7da058 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:56:50.348585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52399559-5a97-4e96-b5ca-7b058c7da058 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:50.349004 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52399559-5a97-4e96-b5ca-7b058c7da058 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:50.349398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52399559-5a97-4e96-b5ca-7b058c7da058 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:50.351095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-52399559-5a97-4e96-b5ca-7b058c7da058 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:50.409178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:56:50.465418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52399559-5a97-4e96-b5ca-7b058c7da058 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:56:50.467029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52399559-5a97-4e96-b5ca-7b058c7da058 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:56:50.486343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 659/1311] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:56:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:56:52.913413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb58d00-fd5c-4da8-baae-3a2820fda86a None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:52.913963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb58d00-fd5c-4da8-baae-3a2820fda86a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:52.913963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb58d00-fd5c-4da8-baae-3a2820fda86a None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:52.915425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 653/1312] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:56:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:56:52.920519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7c0a4c-aeef-4db5-8b7a-0fd4b128dd12 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:52.920947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7c0a4c-aeef-4db5-8b7a-0fd4b128dd12 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:52.921113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7c0a4c-aeef-4db5-8b7a-0fd4b128dd12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:53.006572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-7f7c0a4c-aeef-4db5-8b7a-0fd4b128dd12 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:56:53.007828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7f7c0a4c-aeef-4db5-8b7a-0fd4b128dd12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:53.121535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 660/1313] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:56:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:56:53.163123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb58d00-fd5c-4da8-baae-3a2820fda86a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:56:53.164733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2bb58d00-fd5c-4da8-baae-3a2820fda86a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:53.278776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb58d00-fd5c-4da8-baae-3a2820fda86a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:56:53.360164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:56:53.362728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1859381-cc03-4268-b19e-22c9bb1384e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:56:53.363377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1859381-cc03-4268-b19e-22c9bb1384e0 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:53.363656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1859381-cc03-4268-b19e-22c9bb1384e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:53.363926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1859381-cc03-4268-b19e-22c9bb1384e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:53.366162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b1859381-cc03-4268-b19e-22c9bb1384e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:53.481604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1859381-cc03-4268-b19e-22c9bb1384e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:56:53.483074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1859381-cc03-4268-b19e-22c9bb1384e0 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:56:53.502059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 654/1314] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:56:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:56:56.127183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fbeea6-7b6b-42d9-8d16-a9cffd6ddefc None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:56.127718 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fbeea6-7b6b-42d9-8d16-a9cffd6ddefc None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:56.127837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65fbeea6-7b6b-42d9-8d16-a9cffd6ddefc None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:56.129163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 661/1315] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:56:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:56:56.134770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30b7099-f9bf-42e8-8d5e-4fe90e9d5425 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:56.135146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30b7099-f9bf-42e8-8d5e-4fe90e9d5425 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:56.135324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30b7099-f9bf-42e8-8d5e-4fe90e9d5425 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:56.199039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-f30b7099-f9bf-42e8-8d5e-4fe90e9d5425 None None] MFA Rules not processed for user `5c8b0f52ee8b4d4a82703c331b1da20e`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:56:56.200265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f30b7099-f9bf-42e8-8d5e-4fe90e9d5425 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:56.250150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:56:56.294025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 655/1316] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:56:56 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:56:56.332754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fbeea6-7b6b-42d9-8d16-a9cffd6ddefc None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:56:56.334994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-65fbeea6-7b6b-42d9-8d16-a9cffd6ddefc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:56.344616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:56:56.436819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65fbeea6-7b6b-42d9-8d16-a9cffd6ddefc None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:56:56.507931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa4c2ff1-51d6-476d-b9ed-7680b1626553 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:56:56.508573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4c2ff1-51d6-476d-b9ed-7680b1626553 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:56.508888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4c2ff1-51d6-476d-b9ed-7680b1626553 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:56.509165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa4c2ff1-51d6-476d-b9ed-7680b1626553 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:56.511049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-fa4c2ff1-51d6-476d-b9ed-7680b1626553 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:56.570991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:56:56.615350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4c2ff1-51d6-476d-b9ed-7680b1626553 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:56:56.616812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa4c2ff1-51d6-476d-b9ed-7680b1626553 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:56:56.634621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 662/1317] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:56:56 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:56:59.480696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c4995-8f84-44e5-9a59-726aad9c28de None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:59.481282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c4995-8f84-44e5-9a59-726aad9c28de None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:59.481282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772c4995-8f84-44e5-9a59-726aad9c28de None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:59.482657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 656/1318] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:56:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:56:59.487826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0173a6c-587b-465b-899b-f22f2b5330aa None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:59.488206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0173a6c-587b-465b-899b-f22f2b5330aa None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:59.488414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0173a6c-587b-465b-899b-f22f2b5330aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:59.550310 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-b0173a6c-587b-465b-899b-f22f2b5330aa None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:56:59.551503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b0173a6c-587b-465b-899b-f22f2b5330aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:59.644641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 663/1319] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:56:59 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:56:59.683295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772c4995-8f84-44e5-9a59-726aad9c28de None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:56:59.684853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-772c4995-8f84-44e5-9a59-726aad9c28de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:59.786151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772c4995-8f84-44e5-9a59-726aad9c28de None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:56:59.857104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795a06e2-3d2b-4849-83ef-a17d5671c3c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:56:59.857739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795a06e2-3d2b-4849-83ef-a17d5671c3c3 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:56:59.858017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795a06e2-3d2b-4849-83ef-a17d5671c3c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:56:59.858286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795a06e2-3d2b-4849-83ef-a17d5671c3c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:56:59.859905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-795a06e2-3d2b-4849-83ef-a17d5671c3c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:56:59.866561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:56:59.969661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795a06e2-3d2b-4849-83ef-a17d5671c3c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:56:59.971091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795a06e2-3d2b-4849-83ef-a17d5671c3c3 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:56:59.988877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 657/1320] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:56:59 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:03.025769 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973d5754-8e77-4962-b57a-8e3380d2319f None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:03.026306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973d5754-8e77-4962-b57a-8e3380d2319f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:03.026306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973d5754-8e77-4962-b57a-8e3380d2319f None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:03.033096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a96283-2dfd-4695-889c-d9ae286fd71e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:03.033523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a96283-2dfd-4695-889c-d9ae286fd71e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:03.033649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a96283-2dfd-4695-889c-d9ae286fd71e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:03.036697 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 664/1321] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:03 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:03.100501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-35a96283-2dfd-4695-889c-d9ae286fd71e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:03.101757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-35a96283-2dfd-4695-889c-d9ae286fd71e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:03.161429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:03.211133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 658/1322] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:03.248832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973d5754-8e77-4962-b57a-8e3380d2319f None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:03.249834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-973d5754-8e77-4962-b57a-8e3380d2319f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:03.273668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:03.362128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973d5754-8e77-4962-b57a-8e3380d2319f None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:03.435584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa73f249-cab8-4b59-a6fd-0426929a5745 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:03.436264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa73f249-cab8-4b59-a6fd-0426929a5745 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:03.436577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa73f249-cab8-4b59-a6fd-0426929a5745 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:03.436884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa73f249-cab8-4b59-a6fd-0426929a5745 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:03.438523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-aa73f249-cab8-4b59-a6fd-0426929a5745 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:03.508000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:03.545535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa73f249-cab8-4b59-a6fd-0426929a5745 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:03.546979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa73f249-cab8-4b59-a6fd-0426929a5745 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:03.565987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 665/1323] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 12:57:05.738309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e27969-8a69-4951-bbd3-34dc36896e3f None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:05.738869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e27969-8a69-4951-bbd3-34dc36896e3f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:05.738869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e27969-8a69-4951-bbd3-34dc36896e3f None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:05.740219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 659/1324] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:05.745467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ada2af9-d1cd-40e5-a9ee-347d0043661e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:05.745914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ada2af9-d1cd-40e5-a9ee-347d0043661e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:05.746010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ada2af9-d1cd-40e5-a9ee-347d0043661e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:05.809386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-8ada2af9-d1cd-40e5-a9ee-347d0043661e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:05.810613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8ada2af9-d1cd-40e5-a9ee-347d0043661e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:05.903492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 666/1325] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:05.938604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e27969-8a69-4951-bbd3-34dc36896e3f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:05.940128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-12e27969-8a69-4951-bbd3-34dc36896e3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:06.042109 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e27969-8a69-4951-bbd3-34dc36896e3f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:06.116679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67c4897-9b40-4714-b32e-da1b529e4e73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:06.117358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67c4897-9b40-4714-b32e-da1b529e4e73 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:06.117646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67c4897-9b40-4714-b32e-da1b529e4e73 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:06.117926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67c4897-9b40-4714-b32e-da1b529e4e73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:06.119576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c67c4897-9b40-4714-b32e-da1b529e4e73 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:06.137887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:06.235146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67c4897-9b40-4714-b32e-da1b529e4e73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:06.236604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67c4897-9b40-4714-b32e-da1b529e4e73 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:06.254421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 660/1326] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:08.471901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccedcf4-f204-4d72-9afb-78a56a881427 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:08.472348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccedcf4-f204-4d72-9afb-78a56a881427 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:08.472465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccedcf4-f204-4d72-9afb-78a56a881427 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:08.473925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 667/1327] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:08.479816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c932743-d0ac-4ddb-a0e4-bba77da46864 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:08.480198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c932743-d0ac-4ddb-a0e4-bba77da46864 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:08.480390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c932743-d0ac-4ddb-a0e4-bba77da46864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:08.544677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-7c932743-d0ac-4ddb-a0e4-bba77da46864 None None] MFA Rules not processed for user `5c8b0f52ee8b4d4a82703c331b1da20e`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:08.545916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7c932743-d0ac-4ddb-a0e4-bba77da46864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:08.609833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:08.642117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 661/1328] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:08 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:08.679374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccedcf4-f204-4d72-9afb-78a56a881427 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:08.681003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3ccedcf4-f204-4d72-9afb-78a56a881427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:08.714021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:08.786759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccedcf4-f204-4d72-9afb-78a56a881427 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:08.873843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa470e9a-1e1c-4941-8c00-79648eb188be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:08.874513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa470e9a-1e1c-4941-8c00-79648eb188be service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:08.874803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa470e9a-1e1c-4941-8c00-79648eb188be service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:08.875084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa470e9a-1e1c-4941-8c00-79648eb188be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:08.876864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-fa470e9a-1e1c-4941-8c00-79648eb188be service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:08.958565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:08.991929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa470e9a-1e1c-4941-8c00-79648eb188be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:08.993428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa470e9a-1e1c-4941-8c00-79648eb188be service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:09.012757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 668/1329] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:08 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:11.632257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d44495-a6e2-4199-aad0-60b9364fb780 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:11.632737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d44495-a6e2-4199-aad0-60b9364fb780 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:11.632862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d44495-a6e2-4199-aad0-60b9364fb780 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:11.634198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 662/1330] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:11.639706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d176f8-f331-42ad-9d07-4069c3ed05da None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:11.640113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d176f8-f331-42ad-9d07-4069c3ed05da None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:11.640297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d176f8-f331-42ad-9d07-4069c3ed05da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:11.703318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-60d176f8-f331-42ad-9d07-4069c3ed05da None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:11.704520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-60d176f8-f331-42ad-9d07-4069c3ed05da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:11.799063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 669/1331] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:11.840574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d44495-a6e2-4199-aad0-60b9364fb780 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:11.842209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a1d44495-a6e2-4199-aad0-60b9364fb780 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:11.942822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d44495-a6e2-4199-aad0-60b9364fb780 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:12.013516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582814db-c25e-4574-bf88-0e26da21259f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:12.014155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582814db-c25e-4574-bf88-0e26da21259f service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:12.014462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582814db-c25e-4574-bf88-0e26da21259f service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:12.014741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582814db-c25e-4574-bf88-0e26da21259f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:12.016349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-582814db-c25e-4574-bf88-0e26da21259f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:12.042898 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:12.121979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582814db-c25e-4574-bf88-0e26da21259f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:12.123418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582814db-c25e-4574-bf88-0e26da21259f service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:12.141198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 663/1332] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:14.288433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3ab8a3-9510-4b7d-9470-431943327484 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:14.289023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3ab8a3-9510-4b7d-9470-431943327484 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:14.289023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e3ab8a3-9510-4b7d-9470-431943327484 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:14.290383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 670/1333] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:14.295372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f5d782-1cf8-4f3d-a801-e6564291161e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:14.295746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f5d782-1cf8-4f3d-a801-e6564291161e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:14.295929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f5d782-1cf8-4f3d-a801-e6564291161e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:14.360236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-91f5d782-1cf8-4f3d-a801-e6564291161e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:14.361450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-91f5d782-1cf8-4f3d-a801-e6564291161e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:14.434161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:14.455574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 664/1334] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:14.495168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3ab8a3-9510-4b7d-9470-431943327484 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:14.496677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2e3ab8a3-9510-4b7d-9470-431943327484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:14.537739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:14.599756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e3ab8a3-9510-4b7d-9470-431943327484 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:14.669969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a36295-d924-41ed-b3e8-aed4fad5b28a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:14.670588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a36295-d924-41ed-b3e8-aed4fad5b28a service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:14.670866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a36295-d924-41ed-b3e8-aed4fad5b28a service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:14.671139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a36295-d924-41ed-b3e8-aed4fad5b28a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:14.672755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c9a36295-d924-41ed-b3e8-aed4fad5b28a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:14.751902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:14.777456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a36295-d924-41ed-b3e8-aed4fad5b28a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:14.778893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a36295-d924-41ed-b3e8-aed4fad5b28a service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:14.796919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 671/1335] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:17.886341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd927691-0de8-4eb4-b8a7-d9995b0a82f7 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:17.886935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd927691-0de8-4eb4-b8a7-d9995b0a82f7 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:17.886935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd927691-0de8-4eb4-b8a7-d9995b0a82f7 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:17.888264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 665/1336] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:17.893505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ed6342-ab04-449f-87c4-0ec6bc67f76e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:17.893946 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ed6342-ab04-449f-87c4-0ec6bc67f76e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:17.894073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51ed6342-ab04-449f-87c4-0ec6bc67f76e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:17.956614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-51ed6342-ab04-449f-87c4-0ec6bc67f76e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:17.957835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-51ed6342-ab04-449f-87c4-0ec6bc67f76e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:18.052001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 672/1337] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:17 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:18.100007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15368df-3437-4ff2-a96d-9ddcf2db2f1e None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:18.100291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15368df-3437-4ff2-a96d-9ddcf2db2f1e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:18.100569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15368df-3437-4ff2-a96d-9ddcf2db2f1e None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:18.102141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 666/1338] 10.222.0.25 () {54 vars in 964 bytes} [Fri Jun 10 12:57:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:18.107844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ac8ade-60e1-428d-8dc0-53a3d5c0cd5e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:18.108141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ac8ade-60e1-428d-8dc0-53a3d5c0cd5e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:18.108426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ac8ade-60e1-428d-8dc0-53a3d5c0cd5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:18.162140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:18.182476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-79ac8ade-60e1-428d-8dc0-53a3d5c0cd5e None None] MFA Rules not processed for user `d6794257c90e40bca2c8efd31b0c791a`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:18.184179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-79ac8ade-60e1-428d-8dc0-53a3d5c0cd5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:18.258659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 673/1339] 10.222.0.25 () {58 vars in 1061 bytes} [Fri Jun 10 12:57:18 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:18.267823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15368df-3437-4ff2-a96d-9ddcf2db2f1e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:18.269350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d15368df-3437-4ff2-a96d-9ddcf2db2f1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:18.368936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15368df-3437-4ff2-a96d-9ddcf2db2f1e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:18.439766 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9653d648-e426-4953-8e62-48526a8b6204 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:18.440346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653d648-e426-4953-8e62-48526a8b6204 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:18.440627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653d648-e426-4953-8e62-48526a8b6204 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:18.440933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9653d648-e426-4953-8e62-48526a8b6204 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:18.442554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9653d648-e426-4953-8e62-48526a8b6204 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:18.476700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:18.547612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9653d648-e426-4953-8e62-48526a8b6204 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:18.549098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9653d648-e426-4953-8e62-48526a8b6204 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:18.568677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 667/1340] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:18 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:23.465481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8b7663-3779-4422-a122-4dcfef384caa None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:23.465986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8b7663-3779-4422-a122-4dcfef384caa None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:23.466084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8b7663-3779-4422-a122-4dcfef384caa None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:23.467584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 674/1341] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:23.472593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee07ff7-6702-4feb-92e6-4e81eb4b2737 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:23.473041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee07ff7-6702-4feb-92e6-4e81eb4b2737 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:23.473197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fee07ff7-6702-4feb-92e6-4e81eb4b2737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:23.558228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-fee07ff7-6702-4feb-92e6-4e81eb4b2737 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:23.559442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fee07ff7-6702-4feb-92e6-4e81eb4b2737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:23.652341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:23.664622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 668/1342] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:23.703095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8b7663-3779-4422-a122-4dcfef384caa None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:23.705206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0d8b7663-3779-4422-a122-4dcfef384caa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:23.797037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8b7663-3779-4422-a122-4dcfef384caa None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:23.801105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df72aaac-e9a4-4547-b8af-d5762481d234 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:23.801750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df72aaac-e9a4-4547-b8af-d5762481d234 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:23.802043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df72aaac-e9a4-4547-b8af-d5762481d234 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:23.802321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df72aaac-e9a4-4547-b8af-d5762481d234 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:23.811167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df72aaac-e9a4-4547-b8af-d5762481d234 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:23.812612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df72aaac-e9a4-4547-b8af-d5762481d234 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:23.821954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 675/1343] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:26.902104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b29e95-8b2c-43cb-9cba-d9c05c755503 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:26.902680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b29e95-8b2c-43cb-9cba-d9c05c755503 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:26.902680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b29e95-8b2c-43cb-9cba-d9c05c755503 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:26.904026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 669/1344] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:26.909184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08c7b47-5711-4f5e-a58c-3754bc1aa03c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:26.909593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08c7b47-5711-4f5e-a58c-3754bc1aa03c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:26.909855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08c7b47-5711-4f5e-a58c-3754bc1aa03c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:26.937342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e08c7b47-5711-4f5e-a58c-3754bc1aa03c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:26.938529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e08c7b47-5711-4f5e-a58c-3754bc1aa03c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:26.966035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:27.020623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 676/1345] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:26 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:27.057256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b29e95-8b2c-43cb-9cba-d9c05c755503 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:27.068513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b29e95-8b2c-43cb-9cba-d9c05c755503 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:27.071988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79d12f67-e801-47bb-81a6-d99638ce1dee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:27.072618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d12f67-e801-47bb-81a6-d99638ce1dee service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:27.072936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d12f67-e801-47bb-81a6-d99638ce1dee service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:27.073212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d12f67-e801-47bb-81a6-d99638ce1dee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:27.082248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d12f67-e801-47bb-81a6-d99638ce1dee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:27.083686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79d12f67-e801-47bb-81a6-d99638ce1dee service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:27.093134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 670/1346] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:27 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:31.679284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1cf951-54f3-42bb-9081-90a38e3d15f0 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:31.679779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1cf951-54f3-42bb-9081-90a38e3d15f0 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:31.679779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1cf951-54f3-42bb-9081-90a38e3d15f0 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:31.681480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 677/1347] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:31.686753 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aed7508-d306-488a-9cce-43e35fac572c None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:31.687215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aed7508-d306-488a-9cce-43e35fac572c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:31.687215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aed7508-d306-488a-9cce-43e35fac572c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:31.705346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-2aed7508-d306-488a-9cce-43e35fac572c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:31.706847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2aed7508-d306-488a-9cce-43e35fac572c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:31.715511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 671/1348] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:31.758905 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1cf951-54f3-42bb-9081-90a38e3d15f0 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:31.770802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1cf951-54f3-42bb-9081-90a38e3d15f0 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:31.774364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57167b31-3ede-46e5-9ee3-b4b6c2feb124 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:31.775021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57167b31-3ede-46e5-9ee3-b4b6c2feb124 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:31.775309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57167b31-3ede-46e5-9ee3-b4b6c2feb124 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:31.775713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57167b31-3ede-46e5-9ee3-b4b6c2feb124 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:31.787460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57167b31-3ede-46e5-9ee3-b4b6c2feb124 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:31.788989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57167b31-3ede-46e5-9ee3-b4b6c2feb124 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:31.802828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 678/1349] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:36.990195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c53a9c-3e1f-4c4c-983e-48500d5cd021 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:36.990725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c53a9c-3e1f-4c4c-983e-48500d5cd021 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:36.990837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c53a9c-3e1f-4c4c-983e-48500d5cd021 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:36.992397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 672/1350] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 12:57:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:36.997461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d382672-52eb-4ecd-b11d-60f067132d7d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:36.997866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d382672-52eb-4ecd-b11d-60f067132d7d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:36.998035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d382672-52eb-4ecd-b11d-60f067132d7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:37.073938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5d382672-52eb-4ecd-b11d-60f067132d7d None None] MFA Rules not processed for user `0c315cfb1aa44f3c9a755f3cb8e5cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:37.075152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5d382672-52eb-4ecd-b11d-60f067132d7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:37.197179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 679/1351] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 12:57:36 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:41.342679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf40544-20d8-4fea-8c42-ab8d7b741100 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:41.343141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf40544-20d8-4fea-8c42-ab8d7b741100 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:41.343292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf40544-20d8-4fea-8c42-ab8d7b741100 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:41.344623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 673/1352] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 12:57:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:41.350117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0aacc5-1d65-4a3a-ac2c-196e686a1687 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:41.350580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0aacc5-1d65-4a3a-ac2c-196e686a1687 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:41.350729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0aacc5-1d65-4a3a-ac2c-196e686a1687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:41.370706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6e0aacc5-1d65-4a3a-ac2c-196e686a1687 None None] MFA Rules not processed for user `0c315cfb1aa44f3c9a755f3cb8e5cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:41.372050 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6e0aacc5-1d65-4a3a-ac2c-196e686a1687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:41.381601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 680/1353] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 12:57:41 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:41.699797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=13 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:43.069212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ef6db-5581-4166-8056-93016009c627 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:43.069913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ef6db-5581-4166-8056-93016009c627 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:43.069913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0ef6db-5581-4166-8056-93016009c627 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:43.071170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 674/1354] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 12:57:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:43.076319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d622d75c-87e6-466c-83c5-26285f195cbf None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:43.076754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d622d75c-87e6-466c-83c5-26285f195cbf None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:43.076926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d622d75c-87e6-466c-83c5-26285f195cbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:43.099752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-d622d75c-87e6-466c-83c5-26285f195cbf None None] MFA Rules not processed for user `0c315cfb1aa44f3c9a755f3cb8e5cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:43.100964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d622d75c-87e6-466c-83c5-26285f195cbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:43.109413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 681/1355] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 12:57:43 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:46.852130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e13395d-59c7-4407-a66b-2abdeb81a344 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:46.852726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e13395d-59c7-4407-a66b-2abdeb81a344 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:46.852726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e13395d-59c7-4407-a66b-2abdeb81a344 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:46.854197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 675/1356] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:46.859250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22a0622-bede-4079-b053-ca64e5437c72 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:46.859672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22a0622-bede-4079-b053-ca64e5437c72 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:46.859806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f22a0622-bede-4079-b053-ca64e5437c72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:46.878575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-f22a0622-bede-4079-b053-ca64e5437c72 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:46.880013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f22a0622-bede-4079-b053-ca64e5437c72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:46.893237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 682/1357] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:46.941601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9fc8f1-c21d-4e29-8d94-3f10eeb49570 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:46.942157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9fc8f1-c21d-4e29-8d94-3f10eeb49570 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:46.942653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f9fc8f1-c21d-4e29-8d94-3f10eeb49570 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:46.944362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 676/1358] 10.222.0.25 () {54 vars in 961 bytes} [Fri Jun 10 12:57:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:46.957831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9bf30f-f703-4d79-b82b-0b1f80a30cc4 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:46.958423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9bf30f-f703-4d79-b82b-0b1f80a30cc4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:46.958919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9bf30f-f703-4d79-b82b-0b1f80a30cc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:47.004201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:57:47.013260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3a9bf30f-f703-4d79-b82b-0b1f80a30cc4 None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:47.014566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3a9bf30f-f703-4d79-b82b-0b1f80a30cc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:47.106775 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 683/1359] 10.222.0.25 () {58 vars in 1058 bytes} [Fri Jun 10 12:57:46 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 12:57:47.114716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9fc8f1-c21d-4e29-8d94-3f10eeb49570 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:47.130317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f9fc8f1-c21d-4e29-8d94-3f10eeb49570 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:47.136136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed0984-cdce-4ec7-96a5-ec45ed583096 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:47.140822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed0984-cdce-4ec7-96a5-ec45ed583096 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:47.140822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed0984-cdce-4ec7-96a5-ec45ed583096 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:47.140822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed0984-cdce-4ec7-96a5-ec45ed583096 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:47.146669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ed0984-cdce-4ec7-96a5-ec45ed583096 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:47.148445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ed0984-cdce-4ec7-96a5-ec45ed583096 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:47.158420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 677/1360] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:49.495374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c5802d-023c-4a4e-afc0-7ec7c25ad7b1 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:49.495932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c5802d-023c-4a4e-afc0-7ec7c25ad7b1 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:49.495932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c5802d-023c-4a4e-afc0-7ec7c25ad7b1 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:49.497645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 684/1361] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:49.503573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a601ef6-5673-464d-9a90-f2ceb54463dd None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:49.504124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a601ef6-5673-464d-9a90-f2ceb54463dd None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:49.504124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a601ef6-5673-464d-9a90-f2ceb54463dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:49.523666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-8a601ef6-5673-464d-9a90-f2ceb54463dd None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:49.524868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8a601ef6-5673-464d-9a90-f2ceb54463dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:49.533540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 678/1362] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:49.583289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c5802d-023c-4a4e-afc0-7ec7c25ad7b1 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:49.598837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c5802d-023c-4a4e-afc0-7ec7c25ad7b1 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:49.604314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb20c6fb-3ea0-4edd-aa95-ba3467e9990e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:49.604987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb20c6fb-3ea0-4edd-aa95-ba3467e9990e service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:49.605272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb20c6fb-3ea0-4edd-aa95-ba3467e9990e service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:49.605550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb20c6fb-3ea0-4edd-aa95-ba3467e9990e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:49.614302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb20c6fb-3ea0-4edd-aa95-ba3467e9990e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:49.615923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb20c6fb-3ea0-4edd-aa95-ba3467e9990e service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:49.627708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 685/1363] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 12:57:52.521833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62275e4-c225-414f-afb0-a09f9d7b3336 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:52.522781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62275e4-c225-414f-afb0-a09f9d7b3336 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:52.523291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62275e4-c225-414f-afb0-a09f9d7b3336 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:52.525237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 679/1364] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:52.530945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dc25bf-287d-450a-b650-4c41f2fcb2e1 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:52.531509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dc25bf-287d-450a-b650-4c41f2fcb2e1 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:52.531509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33dc25bf-287d-450a-b650-4c41f2fcb2e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:52.549891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-33dc25bf-287d-450a-b650-4c41f2fcb2e1 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:52.550991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-33dc25bf-287d-450a-b650-4c41f2fcb2e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:52.558966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 686/1365] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:52.604379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9359ce80-67c8-4f5d-9d8a-9d559579e4fa None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:52.604754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9359ce80-67c8-4f5d-9d8a-9d559579e4fa None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:52.605187 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9359ce80-67c8-4f5d-9d8a-9d559579e4fa None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:52.606708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 680/1366] 10.222.0.25 () {54 vars in 961 bytes} [Fri Jun 10 12:57:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:52.612435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6cc689-85c7-4560-ba96-f51cafd6f9b5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:52.616427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6cc689-85c7-4560-ba96-f51cafd6f9b5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:52.616427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6cc689-85c7-4560-ba96-f51cafd6f9b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:52.631428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-2d6cc689-85c7-4560-ba96-f51cafd6f9b5 None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:52.632605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2d6cc689-85c7-4560-ba96-f51cafd6f9b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:52.642142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 687/1367] 10.222.0.25 () {58 vars in 1058 bytes} [Fri Jun 10 12:57:52 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 12:57:52.649664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9359ce80-67c8-4f5d-9d8a-9d559579e4fa None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:52.664370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9359ce80-67c8-4f5d-9d8a-9d559579e4fa None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:52.669732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dead28cb-61a1-4006-b86b-27ffd9402d74 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:52.670367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead28cb-61a1-4006-b86b-27ffd9402d74 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:52.670650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead28cb-61a1-4006-b86b-27ffd9402d74 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:52.670923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dead28cb-61a1-4006-b86b-27ffd9402d74 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:52.679457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dead28cb-61a1-4006-b86b-27ffd9402d74 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:52.681072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dead28cb-61a1-4006-b86b-27ffd9402d74 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:52.690229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 681/1368] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:55.046845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ca2a11-7ad0-4bcf-98a3-7556a990ad1e None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:55.047377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ca2a11-7ad0-4bcf-98a3-7556a990ad1e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:55.047377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ca2a11-7ad0-4bcf-98a3-7556a990ad1e None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:55.048753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 688/1369] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:55.054089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f49dcc4-9a73-4376-8681-77bb1f6a8e04 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:55.054524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f49dcc4-9a73-4376-8681-77bb1f6a8e04 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:55.054644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f49dcc4-9a73-4376-8681-77bb1f6a8e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:55.072895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1f49dcc4-9a73-4376-8681-77bb1f6a8e04 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:55.074078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1f49dcc4-9a73-4376-8681-77bb1f6a8e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:55.082574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 682/1370] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:55.127831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ca2a11-7ad0-4bcf-98a3-7556a990ad1e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:55.144317 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ca2a11-7ad0-4bcf-98a3-7556a990ad1e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:55.150371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c557be-e065-4909-ba72-832e14a7c4e4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:55.151008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c557be-e065-4909-ba72-832e14a7c4e4 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:55.151291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c557be-e065-4909-ba72-832e14a7c4e4 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:55.151565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c557be-e065-4909-ba72-832e14a7c4e4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:55.160160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c557be-e065-4909-ba72-832e14a7c4e4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:55.161785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c557be-e065-4909-ba72-832e14a7c4e4 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:55.169434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7983660-129c-4668-8df2-ecf7beae8488 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:55.169718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7983660-129c-4668-8df2-ecf7beae8488 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:55.170306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7983660-129c-4668-8df2-ecf7beae8488 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:55.171843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 683/1371] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jun 10 12:57:55.171957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 689/1372] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:55.177425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836be2de-9966-4804-a06d-628aad6ad435 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:55.178430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836be2de-9966-4804-a06d-628aad6ad435 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:55.178900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836be2de-9966-4804-a06d-628aad6ad435 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:55.197474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-836be2de-9966-4804-a06d-628aad6ad435 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:55.199661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-836be2de-9966-4804-a06d-628aad6ad435 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:55.207201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 684/1373] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:55.526864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3792fb-9d2c-481c-aab7-71c9b5c26e9b None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:55.527256 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3792fb-9d2c-481c-aab7-71c9b5c26e9b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:55.528432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3792fb-9d2c-481c-aab7-71c9b5c26e9b None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:55.529329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 690/1374] 10.222.0.25 () {54 vars in 963 bytes} [Fri Jun 10 12:57:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:55.534575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc010ef-a0fc-4f6f-9920-a82f6cc041ab None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:55.534875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc010ef-a0fc-4f6f-9920-a82f6cc041ab None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:55.535163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc010ef-a0fc-4f6f-9920-a82f6cc041ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:55.594628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-6fc010ef-a0fc-4f6f-9920-a82f6cc041ab None None] MFA Rules not processed for user `184bf14f33464130ab62ed1853bf4f8d`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:55.595858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6fc010ef-a0fc-4f6f-9920-a82f6cc041ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:55.663137 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 685/1375] 10.222.0.25 () {58 vars in 1060 bytes} [Fri Jun 10 12:57:55 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:55.672939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3792fb-9d2c-481c-aab7-71c9b5c26e9b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:55.683833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3792fb-9d2c-481c-aab7-71c9b5c26e9b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:55.687373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3727f649-f7af-498e-98b9-2ae313aa2df6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:55.688279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727f649-f7af-498e-98b9-2ae313aa2df6 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:55.688566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727f649-f7af-498e-98b9-2ae313aa2df6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:55.688884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3727f649-f7af-498e-98b9-2ae313aa2df6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:55.698884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3727f649-f7af-498e-98b9-2ae313aa2df6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:55.701372 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3727f649-f7af-498e-98b9-2ae313aa2df6 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:55.711962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 691/1376] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:57.551865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d990dc4-be6c-4cd8-ad78-cb3ec0fe3b40 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:57.552406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d990dc4-be6c-4cd8-ad78-cb3ec0fe3b40 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:57.552534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d990dc4-be6c-4cd8-ad78-cb3ec0fe3b40 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:57.553886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 686/1377] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:57.559228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fc39df-47f0-4c8b-8294-74dc284f36f0 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:57.559657 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fc39df-47f0-4c8b-8294-74dc284f36f0 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:57.559882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2fc39df-47f0-4c8b-8294-74dc284f36f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:57.577575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c2fc39df-47f0-4c8b-8294-74dc284f36f0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:57:57.578763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c2fc39df-47f0-4c8b-8294-74dc284f36f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:57:57.587460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 692/1378] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:57:57.631541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d990dc4-be6c-4cd8-ad78-cb3ec0fe3b40 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:57:57.648002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d990dc4-be6c-4cd8-ad78-cb3ec0fe3b40 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:57:57.653531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5512ac00-5a09-4d6c-acba-13f4277e2ecc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:57:57.654176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5512ac00-5a09-4d6c-acba-13f4277e2ecc service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:57.654464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5512ac00-5a09-4d6c-acba-13f4277e2ecc service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:57.654744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5512ac00-5a09-4d6c-acba-13f4277e2ecc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:57.663365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5512ac00-5a09-4d6c-acba-13f4277e2ecc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:57:57.665073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5512ac00-5a09-4d6c-acba-13f4277e2ecc service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:57:57.675110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 687/1379] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:57:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:57:59.989295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910fb60-eac8-40f0-872b-c42e08147ab9 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:59.989862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910fb60-eac8-40f0-872b-c42e08147ab9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:59.989862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7910fb60-eac8-40f0-872b-c42e08147ab9 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:57:59.991206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 693/1380] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:57:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:57:59.996371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323fb8a5-b201-4a4d-a792-e9fb7a58d8f4 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:57:59.996779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323fb8a5-b201-4a4d-a792-e9fb7a58d8f4 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:57:59.996975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323fb8a5-b201-4a4d-a792-e9fb7a58d8f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:00.014979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-323fb8a5-b201-4a4d-a792-e9fb7a58d8f4 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:00.016165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-323fb8a5-b201-4a4d-a792-e9fb7a58d8f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:00.024653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 688/1381] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:57:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:00.067839 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7910fb60-eac8-40f0-872b-c42e08147ab9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:00.082396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7910fb60-eac8-40f0-872b-c42e08147ab9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:00.087853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f416d3cf-cf26-4efb-b7b8-bff408f0d627 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:00.088483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f416d3cf-cf26-4efb-b7b8-bff408f0d627 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:00.088796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f416d3cf-cf26-4efb-b7b8-bff408f0d627 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:00.089070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f416d3cf-cf26-4efb-b7b8-bff408f0d627 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:00.097784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f416d3cf-cf26-4efb-b7b8-bff408f0d627 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:00.099375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f416d3cf-cf26-4efb-b7b8-bff408f0d627 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:00.110371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 694/1382] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:02.409328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbe513c-09b1-45a8-bf84-eea7f7db59fa None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:02.410183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbe513c-09b1-45a8-bf84-eea7f7db59fa None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:02.410465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efbe513c-09b1-45a8-bf84-eea7f7db59fa None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:02.411848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 689/1383] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:58:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:02.417142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b80c715-e870-4c23-b78b-84e769c0cfa4 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:02.417541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b80c715-e870-4c23-b78b-84e769c0cfa4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:02.417729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b80c715-e870-4c23-b78b-84e769c0cfa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:02.436593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-2b80c715-e870-4c23-b78b-84e769c0cfa4 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:02.437932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2b80c715-e870-4c23-b78b-84e769c0cfa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:02.449042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 695/1384] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:58:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:02.487535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbe513c-09b1-45a8-bf84-eea7f7db59fa None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:02.503241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efbe513c-09b1-45a8-bf84-eea7f7db59fa None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:02.509124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d43a545-35f2-494c-b33b-8397f702e6e7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:02.509866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d43a545-35f2-494c-b33b-8397f702e6e7 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:02.510356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d43a545-35f2-494c-b33b-8397f702e6e7 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:02.510846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d43a545-35f2-494c-b33b-8397f702e6e7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:02.520483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d43a545-35f2-494c-b33b-8397f702e6e7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:02.522353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d43a545-35f2-494c-b33b-8397f702e6e7 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:02.532269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 690/1385] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 12:58:05.069823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a967fa-065f-4e74-a9d7-4cc2d4ca9c5c None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:05.070408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a967fa-065f-4e74-a9d7-4cc2d4ca9c5c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:05.070574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a967fa-065f-4e74-a9d7-4cc2d4ca9c5c None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:05.072485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 696/1386] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:58:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:05.078899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee99e25-acb5-435d-9ba9-1e02a3c476b3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:05.079305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee99e25-acb5-435d-9ba9-1e02a3c476b3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:05.079983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee99e25-acb5-435d-9ba9-1e02a3c476b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:05.099066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bee99e25-acb5-435d-9ba9-1e02a3c476b3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:05.100259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bee99e25-acb5-435d-9ba9-1e02a3c476b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:05.108073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 691/1387] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:58:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:05.153565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a967fa-065f-4e74-a9d7-4cc2d4ca9c5c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:05.171787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a967fa-065f-4e74-a9d7-4cc2d4ca9c5c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:05.177290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320ebc4e-e0f1-4091-aa1c-655ffc5b5e5c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:05.177930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320ebc4e-e0f1-4091-aa1c-655ffc5b5e5c service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:05.178217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320ebc4e-e0f1-4091-aa1c-655ffc5b5e5c service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:05.178480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320ebc4e-e0f1-4091-aa1c-655ffc5b5e5c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:05.186902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320ebc4e-e0f1-4091-aa1c-655ffc5b5e5c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:05.188490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320ebc4e-e0f1-4091-aa1c-655ffc5b5e5c service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:05.198058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 697/1388] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:07.612021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa33e84-8b4a-4df4-8373-7c277320f811 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:07.612668 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa33e84-8b4a-4df4-8373-7c277320f811 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:07.612668 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa33e84-8b4a-4df4-8373-7c277320f811 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:07.613998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 692/1389] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:58:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:07.619329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e353b5a-890e-44fb-80b4-fc9a45ea34a8 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:07.619808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e353b5a-890e-44fb-80b4-fc9a45ea34a8 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:07.619972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e353b5a-890e-44fb-80b4-fc9a45ea34a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:07.638420 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5e353b5a-890e-44fb-80b4-fc9a45ea34a8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:07.639621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5e353b5a-890e-44fb-80b4-fc9a45ea34a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:07.647924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 698/1390] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:58:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:07.694021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa33e84-8b4a-4df4-8373-7c277320f811 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:07.709359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa33e84-8b4a-4df4-8373-7c277320f811 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:07.714988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703f11c0-d8a8-4bbb-aae7-1d34ab2252f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:07.715666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703f11c0-d8a8-4bbb-aae7-1d34ab2252f5 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:07.715964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703f11c0-d8a8-4bbb-aae7-1d34ab2252f5 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:07.716259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703f11c0-d8a8-4bbb-aae7-1d34ab2252f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:07.724994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703f11c0-d8a8-4bbb-aae7-1d34ab2252f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:07.726662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703f11c0-d8a8-4bbb-aae7-1d34ab2252f5 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:07.736737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 693/1391] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:10.076163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed956ec-f6a8-4b79-a902-1374428cc35f None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:10.076725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed956ec-f6a8-4b79-a902-1374428cc35f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:10.076725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed956ec-f6a8-4b79-a902-1374428cc35f None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:10.078122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 699/1392] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:58:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:10.083478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c461a49-deab-4c93-b6e4-ac401a46ed3c None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:10.083887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c461a49-deab-4c93-b6e4-ac401a46ed3c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:10.084188 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c461a49-deab-4c93-b6e4-ac401a46ed3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:10.102835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-8c461a49-deab-4c93-b6e4-ac401a46ed3c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:10.104026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8c461a49-deab-4c93-b6e4-ac401a46ed3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:10.112725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 694/1393] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:58:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:10.156580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed956ec-f6a8-4b79-a902-1374428cc35f None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:10.172065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed956ec-f6a8-4b79-a902-1374428cc35f None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:10.178054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f57bc7-6913-49c2-b2dd-5e19f4173133 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:10.178689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f57bc7-6913-49c2-b2dd-5e19f4173133 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:10.178972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f57bc7-6913-49c2-b2dd-5e19f4173133 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:10.179252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f57bc7-6913-49c2-b2dd-5e19f4173133 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:10.187818 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f57bc7-6913-49c2-b2dd-5e19f4173133 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:10.189639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35f57bc7-6913-49c2-b2dd-5e19f4173133 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:10.199018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 700/1394] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:11.385779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2b1d60-a5ed-4161-bc68-172f0fb0ab78 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:11.386286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2b1d60-a5ed-4161-bc68-172f0fb0ab78 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:11.386286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a2b1d60-a5ed-4161-bc68-172f0fb0ab78 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:11.387636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 695/1395] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:58:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:11.393443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd97d8b-ce00-4f2f-950e-ff606ff35228 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:11.393928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd97d8b-ce00-4f2f-950e-ff606ff35228 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:11.394023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd97d8b-ce00-4f2f-950e-ff606ff35228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:11.412892 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-7cd97d8b-ce00-4f2f-950e-ff606ff35228 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:11.414096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7cd97d8b-ce00-4f2f-950e-ff606ff35228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:11.421887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 701/1396] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:58:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:11.478082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e38b64-1dd6-401c-bc51-f8a578011df2 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:11.478376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e38b64-1dd6-401c-bc51-f8a578011df2 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:11.478854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e38b64-1dd6-401c-bc51-f8a578011df2 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:11.480233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 696/1397] 10.222.0.25 () {54 vars in 962 bytes} [Fri Jun 10 12:58:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:11.485195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f1bbf7-c9e1-4276-98e4-c2047dc6ab49 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:11.485527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f1bbf7-c9e1-4276-98e4-c2047dc6ab49 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:11.485811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f1bbf7-c9e1-4276-98e4-c2047dc6ab49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:11.515860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:58:11.534468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-91f1bbf7-c9e1-4276-98e4-c2047dc6ab49 None None] MFA Rules not processed for user `4a83b4eb626343f3aa5b1c9f3fc828fd`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:11.535788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-91f1bbf7-c9e1-4276-98e4-c2047dc6ab49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:11.593964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 702/1398] 10.222.0.25 () {58 vars in 1059 bytes} [Fri Jun 10 12:58:11 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:11.602142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e38b64-1dd6-401c-bc51-f8a578011df2 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:11.613472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e38b64-1dd6-401c-bc51-f8a578011df2 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:11.616988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-330fc378-63a1-4ec8-ac26-3becf7d7203a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a83b4eb626343f3aa5b1c9f3fc828fd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:11.617621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330fc378-63a1-4ec8-ac26-3becf7d7203a service glance] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:11.617903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330fc378-63a1-4ec8-ac26-3becf7d7203a service glance] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:11.618179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-330fc378-63a1-4ec8-ac26-3becf7d7203a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:11.626944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330fc378-63a1-4ec8-ac26-3becf7d7203a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:11.628371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-330fc378-63a1-4ec8-ac26-3becf7d7203a service glance] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:11.638139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 697/1399] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:12.447464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8056c67-34ca-4e71-ad16-220eee3e8f30 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:12.447994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8056c67-34ca-4e71-ad16-220eee3e8f30 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:12.447994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8056c67-34ca-4e71-ad16-220eee3e8f30 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:12.449464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 703/1400] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:58:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:12.454587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8767d19a-897d-4d99-80e7-e6274ec3fe74 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:12.455041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8767d19a-897d-4d99-80e7-e6274ec3fe74 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:12.455249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8767d19a-897d-4d99-80e7-e6274ec3fe74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:12.461227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=93 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:58:12.473836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-8767d19a-897d-4d99-80e7-e6274ec3fe74 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:12.475025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8767d19a-897d-4d99-80e7-e6274ec3fe74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:12.482962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 698/1401] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:58:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:12.496653 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ccb2e8-8c7f-4d27-8458-b5f9e76ab78d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:12.496971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ccb2e8-8c7f-4d27-8458-b5f9e76ab78d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:12.497253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ccb2e8-8c7f-4d27-8458-b5f9e76ab78d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:12.527979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8767d19a-897d-4d99-80e7-e6274ec3fe74 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:12.540696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8767d19a-897d-4d99-80e7-e6274ec3fe74 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:12.546608 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9751a1d-9e2b-4a30-931c-bc3f9ad11b6f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:12.546750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-57ccb2e8-8c7f-4d27-8458-b5f9e76ab78d None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:12.547252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9751a1d-9e2b-4a30-931c-bc3f9ad11b6f service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:12.547533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9751a1d-9e2b-4a30-931c-bc3f9ad11b6f service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:12.547806 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9751a1d-9e2b-4a30-931c-bc3f9ad11b6f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:12.547956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-57ccb2e8-8c7f-4d27-8458-b5f9e76ab78d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:12.556644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9751a1d-9e2b-4a30-931c-bc3f9ad11b6f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:12.558252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9751a1d-9e2b-4a30-931c-bc3f9ad11b6f service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:12.567524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 699/1402] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:12.733962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 704/1403] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 12:58:12 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:12.779088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fae872-3b98-4422-90e2-021e99ea0357 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:12.779533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fae872-3b98-4422-90e2-021e99ea0357 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:12.779955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76fae872-3b98-4422-90e2-021e99ea0357 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:12.781578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 700/1404] 10.222.0.25 () {54 vars in 942 bytes} [Fri Jun 10 12:58:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:12.789197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3a6088-763d-4e95-9ff9-4784e8b0007a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:12.789474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3a6088-763d-4e95-9ff9-4784e8b0007a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:12.789752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3a6088-763d-4e95-9ff9-4784e8b0007a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:12.842414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-aa3a6088-763d-4e95-9ff9-4784e8b0007a None None] MFA Rules not processed for user `bb18cd41c9a14d358aac14ed62ef9e02`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:12.843674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-aa3a6088-763d-4e95-9ff9-4784e8b0007a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:12.888299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:58:12.918395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 705/1405] 10.222.0.25 () {58 vars in 1039 bytes} [Fri Jun 10 12:58:12 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:12.927554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fae872-3b98-4422-90e2-021e99ea0357 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:12.940964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76fae872-3b98-4422-90e2-021e99ea0357 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:12.945785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41a1c0e-ca33-45cc-b7bc-04af80b046b0 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:12.946414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41a1c0e-ca33-45cc-b7bc-04af80b046b0 service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:12.946695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41a1c0e-ca33-45cc-b7bc-04af80b046b0 service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:12.946966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41a1c0e-ca33-45cc-b7bc-04af80b046b0 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:12.955381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41a1c0e-ca33-45cc-b7bc-04af80b046b0 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:12.956968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41a1c0e-ca33-45cc-b7bc-04af80b046b0 service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:12.965066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 701/1406] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 12:58:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 12:58:14.859135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef7d6bb-46d3-42b0-822d-078c12c731d4 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:14.859700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef7d6bb-46d3-42b0-822d-078c12c731d4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:14.859700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fef7d6bb-46d3-42b0-822d-078c12c731d4 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:14.864648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 706/1407] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:58:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:14.866409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09020a8-d86b-4a37-bf09-db685840b2d4 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:14.866833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09020a8-d86b-4a37-bf09-db685840b2d4 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:14.867024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d09020a8-d86b-4a37-bf09-db685840b2d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:14.885590 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-d09020a8-d86b-4a37-bf09-db685840b2d4 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:14.886787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d09020a8-d86b-4a37-bf09-db685840b2d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:14.896946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 702/1408] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:58:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:14.939016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef7d6bb-46d3-42b0-822d-078c12c731d4 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:14.953621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fef7d6bb-46d3-42b0-822d-078c12c731d4 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:14.959847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5773a6b6-3f8e-43a2-a256-11935535a92a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:14.960469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5773a6b6-3f8e-43a2-a256-11935535a92a service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:14.960745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5773a6b6-3f8e-43a2-a256-11935535a92a service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:14.961060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5773a6b6-3f8e-43a2-a256-11935535a92a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:14.969574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5773a6b6-3f8e-43a2-a256-11935535a92a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:14.971184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5773a6b6-3f8e-43a2-a256-11935535a92a service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:14.981570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 707/1409] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:24.880320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=17 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:58:24.976833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=18 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:58:36.370219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa6aa11-be79-43ed-a049-b6287f6196f2 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:36.370747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa6aa11-be79-43ed-a049-b6287f6196f2 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:36.370747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daa6aa11-be79-43ed-a049-b6287f6196f2 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:36.372162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 703/1410] 10.222.0.25 () {54 vars in 918 bytes} [Fri Jun 10 12:58:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:36.377266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7849b1-83a2-4666-90c9-51813a0b45af None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:36.377688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7849b1-83a2-4666-90c9-51813a0b45af None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:36.377819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7849b1-83a2-4666-90c9-51813a0b45af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:36.395425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-af7849b1-83a2-4666-90c9-51813a0b45af None None] MFA Rules not processed for user `0c315cfb1aa44f3c9a755f3cb8e5cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:36.396616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-af7849b1-83a2-4666-90c9-51813a0b45af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:36.404929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 708/1411] 10.222.0.25 () {58 vars in 1015 bytes} [Fri Jun 10 12:58:36 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:36.418895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb6f30b-f7d7-4f60-ae7b-b3c68a9e4510 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:36.419183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb6f30b-f7d7-4f60-ae7b-b3c68a9e4510 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:36.419460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb6f30b-f7d7-4f60-ae7b-b3c68a9e4510 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:36.420813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 704/1412] 10.222.0.25 () {54 vars in 918 bytes} [Fri Jun 10 12:58:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:36.425761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7d0a35-7eb0-4884-86b4-49bd5726a24d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:36.426045 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7d0a35-7eb0-4884-86b4-49bd5726a24d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:36.426320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7d0a35-7eb0-4884-86b4-49bd5726a24d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:36.443217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ae7d0a35-7eb0-4884-86b4-49bd5726a24d None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:36.444512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ae7d0a35-7eb0-4884-86b4-49bd5726a24d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:36.453519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 709/1413] 10.222.0.25 () {58 vars in 1015 bytes} [Fri Jun 10 12:58:36 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 12:58:39.024584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738991f8-46b9-4c93-9818-636aabc2a038 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:39.025176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738991f8-46b9-4c93-9818-636aabc2a038 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:39.025176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738991f8-46b9-4c93-9818-636aabc2a038 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:39.026474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 705/1414] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:58:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:39.031693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4aef2f-1b70-4b39-b1ee-0a4fe89882ce None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:39.032098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4aef2f-1b70-4b39-b1ee-0a4fe89882ce None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:39.032254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4aef2f-1b70-4b39-b1ee-0a4fe89882ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:39.049804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3a4aef2f-1b70-4b39-b1ee-0a4fe89882ce None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:39.050970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3a4aef2f-1b70-4b39-b1ee-0a4fe89882ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:39.058560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 710/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:58:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:39.100882 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738991f8-46b9-4c93-9818-636aabc2a038 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:39.115264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738991f8-46b9-4c93-9818-636aabc2a038 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:39.120492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65b8529d-6d35-477f-b98f-0bd0380d7ccc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:39.121155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b8529d-6d35-477f-b98f-0bd0380d7ccc service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:39.121440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b8529d-6d35-477f-b98f-0bd0380d7ccc service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:39.121735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65b8529d-6d35-477f-b98f-0bd0380d7ccc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:39.130260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b8529d-6d35-477f-b98f-0bd0380d7ccc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:39.131832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65b8529d-6d35-477f-b98f-0bd0380d7ccc service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:39.141299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 706/1416] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:40.080927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c10e366-07fb-4778-adc2-9e48224c521e None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:40.081523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c10e366-07fb-4778-adc2-9e48224c521e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:40.081523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c10e366-07fb-4778-adc2-9e48224c521e None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:40.082874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 711/1417] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 12:58:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:40.087716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4495d7-42dc-494c-b6b1-8f05df0f2c0c None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:40.088137 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4495d7-42dc-494c-b6b1-8f05df0f2c0c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:40.088284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4495d7-42dc-494c-b6b1-8f05df0f2c0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:40.106596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-8d4495d7-42dc-494c-b6b1-8f05df0f2c0c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:40.107901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8d4495d7-42dc-494c-b6b1-8f05df0f2c0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:40.116235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 707/1418] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 12:58:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:40.265580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c10e366-07fb-4778-adc2-9e48224c521e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:40.285374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c10e366-07fb-4778-adc2-9e48224c521e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:40.293137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03bc7bb-8058-44d0-8436-2d5b63aaac0c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:40.294094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03bc7bb-8058-44d0-8436-2d5b63aaac0c service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:40.294520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03bc7bb-8058-44d0-8436-2d5b63aaac0c service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:40.294936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03bc7bb-8058-44d0-8436-2d5b63aaac0c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:40.307798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03bc7bb-8058-44d0-8436-2d5b63aaac0c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:40.310419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03bc7bb-8058-44d0-8436-2d5b63aaac0c service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:40.322212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 712/1419] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:41.237487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0013e41-f862-4480-b528-0ff2a8107dfe None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:41.238080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0013e41-f862-4480-b528-0ff2a8107dfe None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:41.238080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0013e41-f862-4480-b528-0ff2a8107dfe None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:41.239488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 708/1420] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 12:58:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:41.244346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8911c4-368a-4aea-9def-c9ffcc20f9eb None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:41.244793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8911c4-368a-4aea-9def-c9ffcc20f9eb None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:41.244936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8911c4-368a-4aea-9def-c9ffcc20f9eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:41.262984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5f8911c4-368a-4aea-9def-c9ffcc20f9eb None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:41.264155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5f8911c4-368a-4aea-9def-c9ffcc20f9eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:41.271707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 713/1421] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 12:58:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:41.421011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0013e41-f862-4480-b528-0ff2a8107dfe None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:41.435759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0013e41-f862-4480-b528-0ff2a8107dfe None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:41.441143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980e5f22-24b3-4fbd-8409-49d4f2065f03 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:41.441775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e5f22-24b3-4fbd-8409-49d4f2065f03 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:41.442062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e5f22-24b3-4fbd-8409-49d4f2065f03 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:41.442347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980e5f22-24b3-4fbd-8409-49d4f2065f03 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:41.450428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980e5f22-24b3-4fbd-8409-49d4f2065f03 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:41.452011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980e5f22-24b3-4fbd-8409-49d4f2065f03 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:41.461329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 709/1422] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:42.372631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1fe0bc-ea01-41a4-85c0-9cb1ba2757e7 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:42.373109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1fe0bc-ea01-41a4-85c0-9cb1ba2757e7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:42.373269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1fe0bc-ea01-41a4-85c0-9cb1ba2757e7 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:42.375134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 714/1423] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 12:58:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:42.380123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927edf79-a32f-46e2-9bf1-a23b97ac8e95 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:42.380551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927edf79-a32f-46e2-9bf1-a23b97ac8e95 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:42.380690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927edf79-a32f-46e2-9bf1-a23b97ac8e95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:42.399117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-927edf79-a32f-46e2-9bf1-a23b97ac8e95 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:42.400308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-927edf79-a32f-46e2-9bf1-a23b97ac8e95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:42.408081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 710/1424] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 12:58:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:42.556915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1fe0bc-ea01-41a4-85c0-9cb1ba2757e7 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:42.571318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1fe0bc-ea01-41a4-85c0-9cb1ba2757e7 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:42.576520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0baa7c1f-6722-4a06-9741-408bb2466be3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:42.577173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa7c1f-6722-4a06-9741-408bb2466be3 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:42.577470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa7c1f-6722-4a06-9741-408bb2466be3 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:42.577747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0baa7c1f-6722-4a06-9741-408bb2466be3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:42.585603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baa7c1f-6722-4a06-9741-408bb2466be3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:42.587117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0baa7c1f-6722-4a06-9741-408bb2466be3 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:42.596168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 715/1425] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:43.491359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92ff020-aed9-424c-b4ad-acde634c84e6 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:43.491911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92ff020-aed9-424c-b4ad-acde634c84e6 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:43.491911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92ff020-aed9-424c-b4ad-acde634c84e6 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:43.493330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 711/1426] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 12:58:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:43.498095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6341b6cd-98d3-4bd2-bd68-8fe53c76d164 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:43.498493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6341b6cd-98d3-4bd2-bd68-8fe53c76d164 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:43.498643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6341b6cd-98d3-4bd2-bd68-8fe53c76d164 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:43.516853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6341b6cd-98d3-4bd2-bd68-8fe53c76d164 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:43.518025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6341b6cd-98d3-4bd2-bd68-8fe53c76d164 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:43.525610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 716/1427] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 12:58:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:43.675548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92ff020-aed9-424c-b4ad-acde634c84e6 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:43.689985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92ff020-aed9-424c-b4ad-acde634c84e6 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:43.695284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcb247d-2254-463c-be67-41798dac762f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:43.695910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcb247d-2254-463c-be67-41798dac762f service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:43.696195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcb247d-2254-463c-be67-41798dac762f service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:43.696473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcb247d-2254-463c-be67-41798dac762f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:43.704996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcb247d-2254-463c-be67-41798dac762f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:43.706573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcb247d-2254-463c-be67-41798dac762f service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:43.715820 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 712/1428] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:44.676002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe656278-02ad-4a7a-825b-263403010f74 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:44.676532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe656278-02ad-4a7a-825b-263403010f74 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:44.676532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe656278-02ad-4a7a-825b-263403010f74 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:44.677919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 717/1429] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 12:58:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:44.682829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc2561f-2e97-41e1-aec7-58eb92116f08 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:44.683238 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc2561f-2e97-41e1-aec7-58eb92116f08 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:44.683395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc2561f-2e97-41e1-aec7-58eb92116f08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:44.701054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ebc2561f-2e97-41e1-aec7-58eb92116f08 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:44.702241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ebc2561f-2e97-41e1-aec7-58eb92116f08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:44.709925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 713/1430] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 12:58:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:44.859859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe656278-02ad-4a7a-825b-263403010f74 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:44.873897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe656278-02ad-4a7a-825b-263403010f74 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:44.879882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd90eb7-4ddd-4d23-95e1-f387531c53f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:44.880513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd90eb7-4ddd-4d23-95e1-f387531c53f5 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:44.880821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd90eb7-4ddd-4d23-95e1-f387531c53f5 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:44.881107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd90eb7-4ddd-4d23-95e1-f387531c53f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:44.889285 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd90eb7-4ddd-4d23-95e1-f387531c53f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:44.891005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd90eb7-4ddd-4d23-95e1-f387531c53f5 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:44.900219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 718/1431] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:45.815442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb79dd6-6480-456a-a218-2157b3771f11 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:45.815988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb79dd6-6480-456a-a218-2157b3771f11 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:45.815988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb79dd6-6480-456a-a218-2157b3771f11 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:45.817429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 714/1432] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 12:58:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:45.822212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c26bc5-306d-4a8d-8803-9e21c2c33c6a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:45.822631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c26bc5-306d-4a8d-8803-9e21c2c33c6a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:45.822780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c26bc5-306d-4a8d-8803-9e21c2c33c6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:45.840733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-f1c26bc5-306d-4a8d-8803-9e21c2c33c6a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:45.841965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f1c26bc5-306d-4a8d-8803-9e21c2c33c6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:45.850080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 719/1433] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 12:58:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:46.003595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb79dd6-6480-456a-a218-2157b3771f11 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:46.018480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb79dd6-6480-456a-a218-2157b3771f11 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:46.024034 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da88b127-9320-44f7-9e4b-6793c2297681 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:46.024666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da88b127-9320-44f7-9e4b-6793c2297681 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:46.024980 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da88b127-9320-44f7-9e4b-6793c2297681 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:46.025259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da88b127-9320-44f7-9e4b-6793c2297681 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:46.033357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da88b127-9320-44f7-9e4b-6793c2297681 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:46.034952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da88b127-9320-44f7-9e4b-6793c2297681 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:46.044066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 715/1434] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:46.962786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649af4d4-dee7-4309-b051-38cf6fef6776 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:46.963392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649af4d4-dee7-4309-b051-38cf6fef6776 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:46.963392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649af4d4-dee7-4309-b051-38cf6fef6776 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:46.964733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 720/1435] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 12:58:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:46.970172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ed7d5a-9692-4b80-9b2b-7ad250977c51 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:46.970887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ed7d5a-9692-4b80-9b2b-7ad250977c51 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:46.971004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ed7d5a-9692-4b80-9b2b-7ad250977c51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:46.989640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-03ed7d5a-9692-4b80-9b2b-7ad250977c51 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:46.990822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-03ed7d5a-9692-4b80-9b2b-7ad250977c51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:46.998563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 716/1436] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 12:58:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:47.148576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649af4d4-dee7-4309-b051-38cf6fef6776 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:47.163510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649af4d4-dee7-4309-b051-38cf6fef6776 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:47.169080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2035f105-e0be-495c-a838-2b0b616d1dc8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:47.190480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2035f105-e0be-495c-a838-2b0b616d1dc8 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:47.190480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2035f105-e0be-495c-a838-2b0b616d1dc8 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:47.190480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2035f105-e0be-495c-a838-2b0b616d1dc8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:47.190480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2035f105-e0be-495c-a838-2b0b616d1dc8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:47.190480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2035f105-e0be-495c-a838-2b0b616d1dc8 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:47.190480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 721/1437] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:48.169423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d23ad0-34cf-46a6-879d-957d4e97e217 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:48.169992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d23ad0-34cf-46a6-879d-957d4e97e217 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:48.169992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d23ad0-34cf-46a6-879d-957d4e97e217 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:48.171910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 717/1438] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 12:58:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:48.176840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb62a64-4b8c-42fd-924b-faff640968d0 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:48.177266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb62a64-4b8c-42fd-924b-faff640968d0 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:48.177409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb62a64-4b8c-42fd-924b-faff640968d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:48.195741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-9bb62a64-4b8c-42fd-924b-faff640968d0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:48.196970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9bb62a64-4b8c-42fd-924b-faff640968d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:48.204939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 722/1439] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 12:58:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:48.355564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d23ad0-34cf-46a6-879d-957d4e97e217 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:48.370627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d23ad0-34cf-46a6-879d-957d4e97e217 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:48.375950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5655ff7-d822-4169-b710-2cd416849ff1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:48.376583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5655ff7-d822-4169-b710-2cd416849ff1 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:48.376906 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5655ff7-d822-4169-b710-2cd416849ff1 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:48.377189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5655ff7-d822-4169-b710-2cd416849ff1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:48.385380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5655ff7-d822-4169-b710-2cd416849ff1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:48.386965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5655ff7-d822-4169-b710-2cd416849ff1 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:48.396208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 718/1440] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:50.459544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2e4da7-beef-40f5-9907-c693c1f40fae None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:50.460157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2e4da7-beef-40f5-9907-c693c1f40fae None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:50.460157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2e4da7-beef-40f5-9907-c693c1f40fae None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:50.461502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 723/1441] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:58:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:50.466743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0939d77e-1895-49ab-a741-6d3cb3a6c8be None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:50.467159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0939d77e-1895-49ab-a741-6d3cb3a6c8be None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:50.467298 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0939d77e-1895-49ab-a741-6d3cb3a6c8be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:50.485774 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-0939d77e-1895-49ab-a741-6d3cb3a6c8be None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:50.487242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0939d77e-1895-49ab-a741-6d3cb3a6c8be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:50.495205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 719/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:58:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:50.533348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2e4da7-beef-40f5-9907-c693c1f40fae None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:50.543788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2e4da7-beef-40f5-9907-c693c1f40fae None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:50.547757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c643600-6283-4dde-86cb-c0e01c7cabda service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:50.549269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c643600-6283-4dde-86cb-c0e01c7cabda service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:50.549269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c643600-6283-4dde-86cb-c0e01c7cabda service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:50.549269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c643600-6283-4dde-86cb-c0e01c7cabda service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:50.561606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c643600-6283-4dde-86cb-c0e01c7cabda service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:50.563690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c643600-6283-4dde-86cb-c0e01c7cabda service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:50.573267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 724/1443] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:52.823521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab69928-ec05-4ad2-84e5-4a331661a6b6 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:52.824090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab69928-ec05-4ad2-84e5-4a331661a6b6 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:52.824199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab69928-ec05-4ad2-84e5-4a331661a6b6 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:52.825522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 720/1444] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:58:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:52.831199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c3d5b6-eaea-4a63-961d-b644db9aa8be None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:52.831580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c3d5b6-eaea-4a63-961d-b644db9aa8be None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:52.831766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c3d5b6-eaea-4a63-961d-b644db9aa8be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:52.848946 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a1c3d5b6-eaea-4a63-961d-b644db9aa8be None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:52.850129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a1c3d5b6-eaea-4a63-961d-b644db9aa8be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:52.857831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 725/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:58:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:52.895772 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab69928-ec05-4ad2-84e5-4a331661a6b6 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:52.906271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab69928-ec05-4ad2-84e5-4a331661a6b6 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:52.909803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f111785-d84d-44b0-a91b-62878f8b7c1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:52.910435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f111785-d84d-44b0-a91b-62878f8b7c1d service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:52.910717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f111785-d84d-44b0-a91b-62878f8b7c1d service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:52.910988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f111785-d84d-44b0-a91b-62878f8b7c1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:52.919663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f111785-d84d-44b0-a91b-62878f8b7c1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:52.921123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f111785-d84d-44b0-a91b-62878f8b7c1d service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:52.930481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 721/1446] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:55.424723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7b452-70db-425e-98f9-ad20f1e820a9 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:55.425344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7b452-70db-425e-98f9-ad20f1e820a9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:55.425344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e7b452-70db-425e-98f9-ad20f1e820a9 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:55.426657 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 726/1447] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:58:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:55.431892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7800d8-0b62-407d-9948-6caf2eab4f34 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:55.432301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7800d8-0b62-407d-9948-6caf2eab4f34 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:55.432490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7800d8-0b62-407d-9948-6caf2eab4f34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:55.450360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ce7800d8-0b62-407d-9948-6caf2eab4f34 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:55.451550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ce7800d8-0b62-407d-9948-6caf2eab4f34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:55.459365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 722/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:58:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:55.496975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e7b452-70db-425e-98f9-ad20f1e820a9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:55.508358 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e7b452-70db-425e-98f9-ad20f1e820a9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:55.511789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa08c9f4-0a0e-4ac5-a1f3-2b32e5ad6fad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:55.512409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa08c9f4-0a0e-4ac5-a1f3-2b32e5ad6fad service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:55.512691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa08c9f4-0a0e-4ac5-a1f3-2b32e5ad6fad service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:55.513004 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa08c9f4-0a0e-4ac5-a1f3-2b32e5ad6fad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:55.522421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa08c9f4-0a0e-4ac5-a1f3-2b32e5ad6fad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:55.523860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa08c9f4-0a0e-4ac5-a1f3-2b32e5ad6fad service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:55.533251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 727/1449] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:58:57.757303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec23fc-1756-4da2-a6c1-26b89838a082 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:57.757838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec23fc-1756-4da2-a6c1-26b89838a082 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:57.757838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eec23fc-1756-4da2-a6c1-26b89838a082 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:57.759216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 723/1450] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:58:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:58:57.765157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c5683-7587-421b-9992-db6e35a0235b None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:57.765544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c5683-7587-421b-9992-db6e35a0235b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:57.765742 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1c5683-7587-421b-9992-db6e35a0235b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:57.784104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-dd1c5683-7587-421b-9992-db6e35a0235b None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:58:57.785325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-dd1c5683-7587-421b-9992-db6e35a0235b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:58:57.792977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 728/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:58:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:58:57.830613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec23fc-1756-4da2-a6c1-26b89838a082 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:58:57.841085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eec23fc-1756-4da2-a6c1-26b89838a082 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:58:57.845127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733d6589-efb0-4015-add2-a803d832c784 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:58:57.845747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733d6589-efb0-4015-add2-a803d832c784 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:58:57.846025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733d6589-efb0-4015-add2-a803d832c784 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:58:57.846299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733d6589-efb0-4015-add2-a803d832c784 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:58:57.854847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733d6589-efb0-4015-add2-a803d832c784 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:58:57.856265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-733d6589-efb0-4015-add2-a803d832c784 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:58:57.865488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 724/1452] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:58:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:00.051079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d56733-64c8-442f-b69a-f0125aaec8d7 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:00.051643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d56733-64c8-442f-b69a-f0125aaec8d7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:00.051643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d56733-64c8-442f-b69a-f0125aaec8d7 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:00.053014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 729/1453] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:59:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:00.058085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60499a57-fab3-46c4-aacd-1885bb6bb811 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:00.058502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60499a57-fab3-46c4-aacd-1885bb6bb811 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:00.058682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60499a57-fab3-46c4-aacd-1885bb6bb811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:00.076726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-60499a57-fab3-46c4-aacd-1885bb6bb811 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:00.078018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-60499a57-fab3-46c4-aacd-1885bb6bb811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:00.085833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 725/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:59:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:00.122885 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d56733-64c8-442f-b69a-f0125aaec8d7 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:00.134206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d56733-64c8-442f-b69a-f0125aaec8d7 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:00.137611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a835fa4-5654-4737-9c52-df3c1500198b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:00.138291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a835fa4-5654-4737-9c52-df3c1500198b service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:00.138566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a835fa4-5654-4737-9c52-df3c1500198b service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:00.138834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a835fa4-5654-4737-9c52-df3c1500198b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:00.147351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a835fa4-5654-4737-9c52-df3c1500198b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:00.148786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a835fa4-5654-4737-9c52-df3c1500198b service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:00.158267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 730/1455] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:02.379036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43302a2a-a20d-44cd-8c13-c27f9ce43e71 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:02.379548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43302a2a-a20d-44cd-8c13-c27f9ce43e71 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:02.379672 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43302a2a-a20d-44cd-8c13-c27f9ce43e71 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:02.381241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 726/1456] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:59:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:02.386562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12f214-7b5c-4a4a-b0f7-0f353a6d6af2 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:02.387022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12f214-7b5c-4a4a-b0f7-0f353a6d6af2 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:02.387133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af12f214-7b5c-4a4a-b0f7-0f353a6d6af2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:02.404162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-af12f214-7b5c-4a4a-b0f7-0f353a6d6af2 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:02.405907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-af12f214-7b5c-4a4a-b0f7-0f353a6d6af2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:02.413576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 731/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:59:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:02.451947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43302a2a-a20d-44cd-8c13-c27f9ce43e71 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:02.462440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43302a2a-a20d-44cd-8c13-c27f9ce43e71 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:02.465910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf59dd9-a16d-40a7-9769-0140a228ce04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:02.466538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf59dd9-a16d-40a7-9769-0140a228ce04 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:02.466824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf59dd9-a16d-40a7-9769-0140a228ce04 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:02.467604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf59dd9-a16d-40a7-9769-0140a228ce04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:02.476236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf59dd9-a16d-40a7-9769-0140a228ce04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:02.477691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf59dd9-a16d-40a7-9769-0140a228ce04 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:02.487463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 727/1458] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:04.638172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f0ba11-d416-49af-8315-5393f003d934 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:04.639229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f0ba11-d416-49af-8315-5393f003d934 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:04.639229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f0ba11-d416-49af-8315-5393f003d934 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:04.640585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 732/1459] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:59:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:04.646151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c65e87-69a8-4eef-b127-83ca4089e4e8 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:04.646599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c65e87-69a8-4eef-b127-83ca4089e4e8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:04.646716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c65e87-69a8-4eef-b127-83ca4089e4e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:04.665003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-89c65e87-69a8-4eef-b127-83ca4089e4e8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:04.666403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-89c65e87-69a8-4eef-b127-83ca4089e4e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:04.674719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 728/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:59:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:04.712226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f0ba11-d416-49af-8315-5393f003d934 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:04.723043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f0ba11-d416-49af-8315-5393f003d934 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:04.726585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a4cfdc-c96b-4d09-b969-202b02e02f1d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:04.727212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a4cfdc-c96b-4d09-b969-202b02e02f1d service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:04.727492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a4cfdc-c96b-4d09-b969-202b02e02f1d service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:04.727771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a4cfdc-c96b-4d09-b969-202b02e02f1d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:04.736592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a4cfdc-c96b-4d09-b969-202b02e02f1d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:04.738059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a4cfdc-c96b-4d09-b969-202b02e02f1d service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:04.747382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 733/1461] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:07.015570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4278b38f-1621-418a-a7e7-fdba06bf53df None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:07.016162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4278b38f-1621-418a-a7e7-fdba06bf53df None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:07.016162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4278b38f-1621-418a-a7e7-fdba06bf53df None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:07.017565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 729/1462] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:59:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:07.022604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a22d4a-2fc0-4354-9a49-fac4c8f3ac28 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:07.022988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a22d4a-2fc0-4354-9a49-fac4c8f3ac28 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:07.023156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a22d4a-2fc0-4354-9a49-fac4c8f3ac28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:07.029605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=79 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:59:07.041228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-d8a22d4a-2fc0-4354-9a49-fac4c8f3ac28 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:07.042397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d8a22d4a-2fc0-4354-9a49-fac4c8f3ac28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:07.050539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 734/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:59:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:07.086617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4278b38f-1621-418a-a7e7-fdba06bf53df None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:07.092557 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=73 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:59:07.098354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4278b38f-1621-418a-a7e7-fdba06bf53df None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:07.101909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ad2f73-5b62-4b68-90a3-e4f53e0f5f2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:07.102546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad2f73-5b62-4b68-90a3-e4f53e0f5f2b service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:07.102833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad2f73-5b62-4b68-90a3-e4f53e0f5f2b service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:07.103112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ad2f73-5b62-4b68-90a3-e4f53e0f5f2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:07.112694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ad2f73-5b62-4b68-90a3-e4f53e0f5f2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:07.114165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ad2f73-5b62-4b68-90a3-e4f53e0f5f2b service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:07.123548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 730/1464] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:09.241580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad7d2e8-1581-4337-9e08-01225b0aa2d7 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:09.242157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad7d2e8-1581-4337-9e08-01225b0aa2d7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:09.242157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad7d2e8-1581-4337-9e08-01225b0aa2d7 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:09.243474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 735/1465] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:09.248586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d07a2b-5cd9-4989-b9d5-f32264f6a194 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:09.249025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d07a2b-5cd9-4989-b9d5-f32264f6a194 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:09.249243 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d07a2b-5cd9-4989-b9d5-f32264f6a194 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:09.266761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-27d07a2b-5cd9-4989-b9d5-f32264f6a194 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:09.267984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-27d07a2b-5cd9-4989-b9d5-f32264f6a194 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:09.275729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 731/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:59:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:09.311846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad7d2e8-1581-4337-9e08-01225b0aa2d7 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:09.322467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad7d2e8-1581-4337-9e08-01225b0aa2d7 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:09.326040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f54c8c3-9ce9-446d-96c4-1e7712e6ab1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:09.326668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f54c8c3-9ce9-446d-96c4-1e7712e6ab1e service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:09.326944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f54c8c3-9ce9-446d-96c4-1e7712e6ab1e service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:09.327213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f54c8c3-9ce9-446d-96c4-1e7712e6ab1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:09.336295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f54c8c3-9ce9-446d-96c4-1e7712e6ab1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:09.337764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f54c8c3-9ce9-446d-96c4-1e7712e6ab1e service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:09.347566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 736/1467] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:11.489758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7645be2-7a35-4716-aaee-ae85e8cfb24f None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:11.490287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7645be2-7a35-4716-aaee-ae85e8cfb24f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:11.490381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7645be2-7a35-4716-aaee-ae85e8cfb24f None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:11.491698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 732/1468] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 12:59:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:11.497041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76db722e-4774-4097-8aee-af75c13e969e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:11.497446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76db722e-4774-4097-8aee-af75c13e969e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:11.497591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76db722e-4774-4097-8aee-af75c13e969e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:11.516290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-76db722e-4774-4097-8aee-af75c13e969e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:11.517561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-76db722e-4774-4097-8aee-af75c13e969e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:11.525143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 737/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 12:59:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 12:59:11.560107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7645be2-7a35-4716-aaee-ae85e8cfb24f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:11.571215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7645be2-7a35-4716-aaee-ae85e8cfb24f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:11.574691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed2884b-bdc6-43f6-9c5e-a9a6f6fc88f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:11.575326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed2884b-bdc6-43f6-9c5e-a9a6f6fc88f7 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:11.575613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed2884b-bdc6-43f6-9c5e-a9a6f6fc88f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:11.575936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed2884b-bdc6-43f6-9c5e-a9a6f6fc88f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:11.585439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed2884b-bdc6-43f6-9c5e-a9a6f6fc88f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:11.586921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed2884b-bdc6-43f6-9c5e-a9a6f6fc88f7 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:11.596323 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 733/1470] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:21.510871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=11 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:59:21.592153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=14 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:59:31.480227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3042cef-cf40-40c7-a549-7c32619ccf54 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:31.480799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3042cef-cf40-40c7-a549-7c32619ccf54 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:31.480799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3042cef-cf40-40c7-a549-7c32619ccf54 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:31.482684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 738/1471] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:59:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:31.487730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3090a104-1112-4773-b777-f52e190166de None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:31.488164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3090a104-1112-4773-b777-f52e190166de None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:31.488296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3090a104-1112-4773-b777-f52e190166de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:31.507160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3090a104-1112-4773-b777-f52e190166de None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:31.508402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3090a104-1112-4773-b777-f52e190166de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:31.516208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 734/1472] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:59:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:31.563550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5a55fe-53e6-4b91-a311-e8a1f1919a6b None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:31.563838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5a55fe-53e6-4b91-a311-e8a1f1919a6b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:31.564114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f5a55fe-53e6-4b91-a311-e8a1f1919a6b None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:31.565486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 739/1473] 10.222.0.25 () {54 vars in 962 bytes} [Fri Jun 10 12:59:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:31.570173 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dbca1-e205-4ed5-afd5-4b8d5c8a5e1b None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:31.570469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dbca1-e205-4ed5-afd5-4b8d5c8a5e1b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:31.570750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49dbca1-e205-4ed5-afd5-4b8d5c8a5e1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:31.588128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e49dbca1-e205-4ed5-afd5-4b8d5c8a5e1b None None] MFA Rules not processed for user `4a83b4eb626343f3aa5b1c9f3fc828fd`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:31.589334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e49dbca1-e205-4ed5-afd5-4b8d5c8a5e1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:31.595695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 735/1474] 10.222.0.25 () {58 vars in 1059 bytes} [Fri Jun 10 12:59:31 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 26 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:31.602717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5a55fe-53e6-4b91-a311-e8a1f1919a6b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:31.613825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f5a55fe-53e6-4b91-a311-e8a1f1919a6b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:31.617752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56e39b6-630b-42c2-b9d8-ea0456e98eff service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a83b4eb626343f3aa5b1c9f3fc828fd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:31.618370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e39b6-630b-42c2-b9d8-ea0456e98eff service glance] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:31.618652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e39b6-630b-42c2-b9d8-ea0456e98eff service glance] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:31.618928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56e39b6-630b-42c2-b9d8-ea0456e98eff service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:31.628217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56e39b6-630b-42c2-b9d8-ea0456e98eff service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:31.629665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56e39b6-630b-42c2-b9d8-ea0456e98eff service glance] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:31.639658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 740/1475] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:33.850439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72142b44-bd76-441b-a989-12f7e1306c05 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:33.851003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72142b44-bd76-441b-a989-12f7e1306c05 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:33.851003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72142b44-bd76-441b-a989-12f7e1306c05 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:33.852352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 736/1476] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:59:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:33.857605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b262231-7ee0-47cd-9f69-0791e8ecbc98 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:33.858001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b262231-7ee0-47cd-9f69-0791e8ecbc98 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:33.858179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b262231-7ee0-47cd-9f69-0791e8ecbc98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:33.875039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5b262231-7ee0-47cd-9f69-0791e8ecbc98 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:33.876212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5b262231-7ee0-47cd-9f69-0791e8ecbc98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:33.884472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 741/1477] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:59:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:33.908087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72142b44-bd76-441b-a989-12f7e1306c05 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:33.920423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72142b44-bd76-441b-a989-12f7e1306c05 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:33.925141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbfdcde-a577-4aad-bb17-b3af476a9484 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:33.925850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbfdcde-a577-4aad-bb17-b3af476a9484 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:33.926132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbfdcde-a577-4aad-bb17-b3af476a9484 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:33.926411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbfdcde-a577-4aad-bb17-b3af476a9484 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:33.927142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbfdcde-a577-4aad-bb17-b3af476a9484 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:33.928989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING py.warnings [None req-8fbfdcde-a577-4aad-bb17-b3af476a9484 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 12:59:33.928989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: warnings.warn(msg) Jun 10 12:59:33.928989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: Jun 10 12:59:33.930165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbfdcde-a577-4aad-bb17-b3af476a9484 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:33.950271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 737/1478] 10.222.0.25 () {56 vars in 1108 bytes} [Fri Jun 10 12:59:33 2022] GET /identity/v3/projects => generated 2553 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 12:59:36.066303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67f495a-cea8-4551-be07-d907d0ac2a76 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:36.066866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67f495a-cea8-4551-be07-d907d0ac2a76 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:36.066866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67f495a-cea8-4551-be07-d907d0ac2a76 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:36.068225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 742/1479] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:59:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:36.073348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0b77f4-cfb9-4971-bad9-a97fef2ea0c9 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:36.073789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0b77f4-cfb9-4971-bad9-a97fef2ea0c9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:36.073918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0b77f4-cfb9-4971-bad9-a97fef2ea0c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:36.092787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1f0b77f4-cfb9-4971-bad9-a97fef2ea0c9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:36.093985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1f0b77f4-cfb9-4971-bad9-a97fef2ea0c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:36.101702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 738/1480] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:59:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:36.144675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67f495a-cea8-4551-be07-d907d0ac2a76 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:36.159070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67f495a-cea8-4551-be07-d907d0ac2a76 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:36.164412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2db7e49-393a-4d1e-968f-f945c68f634f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:36.165064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db7e49-393a-4d1e-968f-f945c68f634f service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:36.165343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db7e49-393a-4d1e-968f-f945c68f634f service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:36.165621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2db7e49-393a-4d1e-968f-f945c68f634f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:36.173491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2db7e49-393a-4d1e-968f-f945c68f634f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:36.175082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2db7e49-393a-4d1e-968f-f945c68f634f service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:36.184670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 743/1481] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:38.490398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86f8152-7d01-47f1-9265-3b5abae9b8d5 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:38.490957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86f8152-7d01-47f1-9265-3b5abae9b8d5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:38.490957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86f8152-7d01-47f1-9265-3b5abae9b8d5 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:38.492363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 739/1482] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:59:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:38.497615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faa826-bc1f-45c6-9b24-950f46db9d99 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:38.498025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faa826-bc1f-45c6-9b24-950f46db9d99 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:38.498167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21faa826-bc1f-45c6-9b24-950f46db9d99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:38.515380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-21faa826-bc1f-45c6-9b24-950f46db9d99 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:38.516541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-21faa826-bc1f-45c6-9b24-950f46db9d99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:38.524082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 744/1483] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:59:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:38.560199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86f8152-7d01-47f1-9265-3b5abae9b8d5 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:38.570950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86f8152-7d01-47f1-9265-3b5abae9b8d5 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:38.574502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adf8e04-fefb-48bd-be33-b9958e129ddf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:38.579950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adf8e04-fefb-48bd-be33-b9958e129ddf service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:38.579950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adf8e04-fefb-48bd-be33-b9958e129ddf service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:38.579950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adf8e04-fefb-48bd-be33-b9958e129ddf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:38.585061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adf8e04-fefb-48bd-be33-b9958e129ddf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:38.586494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3adf8e04-fefb-48bd-be33-b9958e129ddf service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:38.596696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 740/1484] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:40.762699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d50c97f-7d7c-4950-89e6-ab7ac886b719 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:40.763239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d50c97f-7d7c-4950-89e6-ab7ac886b719 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:40.763239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d50c97f-7d7c-4950-89e6-ab7ac886b719 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:40.764585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 745/1485] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:59:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:40.769929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dbb813-14b3-4cc2-b243-9622c4dd5ca3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:40.770360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dbb813-14b3-4cc2-b243-9622c4dd5ca3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:40.770494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82dbb813-14b3-4cc2-b243-9622c4dd5ca3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:40.788554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-82dbb813-14b3-4cc2-b243-9622c4dd5ca3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:40.789805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-82dbb813-14b3-4cc2-b243-9622c4dd5ca3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:40.802095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 741/1486] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:59:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:40.840460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d50c97f-7d7c-4950-89e6-ab7ac886b719 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:40.851082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d50c97f-7d7c-4950-89e6-ab7ac886b719 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:40.854615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c2e537-7daf-46c3-bfe2-bb1d6e942d0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:40.855245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c2e537-7daf-46c3-bfe2-bb1d6e942d0a service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:40.856065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c2e537-7daf-46c3-bfe2-bb1d6e942d0a service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:40.856344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c2e537-7daf-46c3-bfe2-bb1d6e942d0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:40.865164 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c2e537-7daf-46c3-bfe2-bb1d6e942d0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:40.866587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c2e537-7daf-46c3-bfe2-bb1d6e942d0a service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:40.875754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 746/1487] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:43.079475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c0f23b-7780-4d31-836e-e278905a7f63 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:43.080045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c0f23b-7780-4d31-836e-e278905a7f63 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:43.080045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c0f23b-7780-4d31-836e-e278905a7f63 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:43.081436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 742/1488] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:59:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:43.086687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfddcbc8-d94d-422a-b82d-1d0b84e8756f None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:43.088075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfddcbc8-d94d-422a-b82d-1d0b84e8756f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:43.088075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfddcbc8-d94d-422a-b82d-1d0b84e8756f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:43.106767 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-dfddcbc8-d94d-422a-b82d-1d0b84e8756f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:43.107944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-dfddcbc8-d94d-422a-b82d-1d0b84e8756f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:43.115573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 747/1489] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:59:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:43.156417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c0f23b-7780-4d31-836e-e278905a7f63 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:43.166978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c0f23b-7780-4d31-836e-e278905a7f63 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:43.170469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a8ff9c-232d-47eb-9947-5df7cb37817f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:43.171090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a8ff9c-232d-47eb-9947-5df7cb37817f service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:43.171370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a8ff9c-232d-47eb-9947-5df7cb37817f service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:43.171646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a8ff9c-232d-47eb-9947-5df7cb37817f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:43.180250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a8ff9c-232d-47eb-9947-5df7cb37817f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:43.181704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a8ff9c-232d-47eb-9947-5df7cb37817f service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:43.191347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 743/1490] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:45.854204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3b01c3-7cc3-47e7-a6d8-5e8c2f867712 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:45.854780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3b01c3-7cc3-47e7-a6d8-5e8c2f867712 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:45.854780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3b01c3-7cc3-47e7-a6d8-5e8c2f867712 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:45.856633 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 748/1491] 10.222.0.25 () {54 vars in 926 bytes} [Fri Jun 10 12:59:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 12:59:45.861759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067036a9-5b02-4ca3-98a1-abad0d9a6870 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:45.862167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067036a9-5b02-4ca3-98a1-abad0d9a6870 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:45.862324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067036a9-5b02-4ca3-98a1-abad0d9a6870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:45.880341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-067036a9-5b02-4ca3-98a1-abad0d9a6870 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 12:59:45.881561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-067036a9-5b02-4ca3-98a1-abad0d9a6870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:45.889359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 744/1492] 10.222.0.25 () {58 vars in 1023 bytes} [Fri Jun 10 12:59:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 12:59:45.931437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3b01c3-7cc3-47e7-a6d8-5e8c2f867712 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 12:59:45.933208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2c3b01c3-7cc3-47e7-a6d8-5e8c2f867712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 12:59:45.945773 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3b01c3-7cc3-47e7-a6d8-5e8c2f867712 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 12:59:45.949344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45caa65d-8932-4302-91ad-03c6fad40ca7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 12:59:45.949978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45caa65d-8932-4302-91ad-03c6fad40ca7 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 12:59:45.950260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45caa65d-8932-4302-91ad-03c6fad40ca7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 12:59:45.950537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45caa65d-8932-4302-91ad-03c6fad40ca7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 12:59:45.959561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45caa65d-8932-4302-91ad-03c6fad40ca7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 12:59:45.961018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45caa65d-8932-4302-91ad-03c6fad40ca7 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 12:59:45.971018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 749/1493] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 12:59:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 12:59:55.875116 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=33 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 12:59:55.966777 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=33 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:01:21.108206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7a2464-9ac6-49f4-b69d-20fe1da46609 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.108886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7a2464-9ac6-49f4-b69d-20fe1da46609 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.108886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7a2464-9ac6-49f4-b69d-20fe1da46609 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.126612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-0c7a2464-9ac6-49f4-b69d-20fe1da46609 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:01:21.127805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0c7a2464-9ac6-49f4-b69d-20fe1da46609 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:01:21.135979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 745/1494] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:01:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:01:21.145051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ecb111-6dff-4d52-8636-c4ddd4782a75 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.145528 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ecb111-6dff-4d52-8636-c4ddd4782a75 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.145632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ecb111-6dff-4d52-8636-c4ddd4782a75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.163341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-96ecb111-6dff-4d52-8636-c4ddd4782a75 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:01:21.164514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-96ecb111-6dff-4d52-8636-c4ddd4782a75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:01:21.172205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 750/1495] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:01:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:01:21.176690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7a2464-9ac6-49f4-b69d-20fe1da46609 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:21.186814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7a2464-9ac6-49f4-b69d-20fe1da46609 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:21.191604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:21.192280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.192565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.192865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.193602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:01:21.194558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:21.195601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING py.warnings [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:01:21.195601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: warnings.warn(msg) Jun 10 13:01:21.195601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: Jun 10 13:01:21.196493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:21.210240 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 746/1496] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:01:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:01:21.221088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ecb111-6dff-4d52-8636-c4ddd4782a75 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:21.230665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96ecb111-6dff-4d52-8636-c4ddd4782a75 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:21.235544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104fd8e0-2382-4d20-b2dc-1653fa0050f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:21.236208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104fd8e0-2382-4d20-b2dc-1653fa0050f7 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.236494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104fd8e0-2382-4d20-b2dc-1653fa0050f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.236794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104fd8e0-2382-4d20-b2dc-1653fa0050f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.237488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104fd8e0-2382-4d20-b2dc-1653fa0050f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:21.239240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WARNING py.warnings [None req-104fd8e0-2382-4d20-b2dc-1653fa0050f7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:01:21.239240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: warnings.warn(msg) Jun 10 13:01:21.239240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: Jun 10 13:01:21.240446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-104fd8e0-2382-4d20-b2dc-1653fa0050f7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:21.363342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 751/1497] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:01:21 2022] POST /identity/v3/projects => generated 377 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:01:21.368294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:21.378155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-becb6fa7-0247-4a87-817d-7f72ab8cd6f3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:21.433010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc316e02-576a-4a97-a862-f06ec628ed52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:21.433675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc316e02-576a-4a97-a862-f06ec628ed52 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.433961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc316e02-576a-4a97-a862-f06ec628ed52 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.434241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc316e02-576a-4a97-a862-f06ec628ed52 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.434932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc316e02-576a-4a97-a862-f06ec628ed52 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:21.435947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING py.warnings [None req-fc316e02-576a-4a97-a862-f06ec628ed52 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:01:21.435947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: warnings.warn(msg) Jun 10 13:01:21.435947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: Jun 10 13:01:21.437189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc316e02-576a-4a97-a862-f06ec628ed52 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:21.513756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 747/1498] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:01:21 2022] POST /identity/v3/users => generated 289 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:01:21.519799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104fd8e0-2382-4d20-b2dc-1653fa0050f7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:21.528118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-104fd8e0-2382-4d20-b2dc-1653fa0050f7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:21.534228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06758806-a5dc-4e6e-b7eb-bc56e71b74a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:21.535424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06758806-a5dc-4e6e-b7eb-bc56e71b74a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.535888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06758806-a5dc-4e6e-b7eb-bc56e71b74a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.536352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06758806-a5dc-4e6e-b7eb-bc56e71b74a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.537148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06758806-a5dc-4e6e-b7eb-bc56e71b74a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:21.538689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WARNING py.warnings [None req-06758806-a5dc-4e6e-b7eb-bc56e71b74a9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:01:21.538689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: warnings.warn(msg) Jun 10 13:01:21.538689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: Jun 10 13:01:21.539635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06758806-a5dc-4e6e-b7eb-bc56e71b74a9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:21.557440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 752/1499] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:01:21.561637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc316e02-576a-4a97-a862-f06ec628ed52 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:21.571634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc316e02-576a-4a97-a862-f06ec628ed52 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:21.576179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca5a5c1-a741-413c-9deb-f566a7840f70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:21.576871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5a5c1-a741-413c-9deb-f566a7840f70 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.577172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5a5c1-a741-413c-9deb-f566a7840f70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.577450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca5a5c1-a741-413c-9deb-f566a7840f70 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.578177 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca5a5c1-a741-413c-9deb-f566a7840f70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:21.579192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING py.warnings [None req-2ca5a5c1-a741-413c-9deb-f566a7840f70 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:01:21.579192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: warnings.warn(msg) Jun 10 13:01:21.579192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: Jun 10 13:01:21.580100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ca5a5c1-a741-413c-9deb-f566a7840f70 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:21.593504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 748/1500] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:01:21.597829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06758806-a5dc-4e6e-b7eb-bc56e71b74a9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:21.608127 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06758806-a5dc-4e6e-b7eb-bc56e71b74a9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:21.612535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea3354cc-643f-4bca-af67-e553064ad83d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:21.613492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3354cc-643f-4bca-af67-e553064ad83d admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.613727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3354cc-643f-4bca-af67-e553064ad83d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.614003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea3354cc-643f-4bca-af67-e553064ad83d admin admin] PATH_INFO: `/v3/projects/e0cc6fc3cde240e689739878ec42e699/users/769c5b2dea984d47b43c9b0749ab8b00/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.629120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3354cc-643f-4bca-af67-e553064ad83d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0cc6fc3cde240e689739878ec42e699, user_id=769c5b2dea984d47b43c9b0749ab8b00, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:21.630150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WARNING py.warnings [None req-ea3354cc-643f-4bca-af67-e553064ad83d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:01:21.630150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: warnings.warn(msg) Jun 10 13:01:21.630150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: Jun 10 13:01:21.632087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea3354cc-643f-4bca-af67-e553064ad83d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:21.704153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 753/1501] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:01:21 2022] PUT /identity/v3/projects/e0cc6fc3cde240e689739878ec42e699/users/769c5b2dea984d47b43c9b0749ab8b00/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:01:21.715414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea98b0c-d4f4-4a08-a1d3-87ea15164bd0 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.718828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea98b0c-d4f4-4a08-a1d3-87ea15164bd0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.718828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea98b0c-d4f4-4a08-a1d3-87ea15164bd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.742891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-dea98b0c-d4f4-4a08-a1d3-87ea15164bd0 None None] MFA Rules not processed for user `769c5b2dea984d47b43c9b0749ab8b00`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:01:21.744070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-dea98b0c-d4f4-4a08-a1d3-87ea15164bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:01:21.806461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 749/1502] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:01:21 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:01:21.820552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b48654b-8283-452d-a0b9-c131a2162b4c None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.820882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b48654b-8283-452d-a0b9-c131a2162b4c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.821168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b48654b-8283-452d-a0b9-c131a2162b4c None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.822631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 754/1503] 10.222.0.25 () {54 vars in 963 bytes} [Fri Jun 10 13:01:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:01:21.827623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5d69f6-73f5-4aa5-a847-2975f743c8ea None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.828032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5d69f6-73f5-4aa5-a847-2975f743c8ea None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.828320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5d69f6-73f5-4aa5-a847-2975f743c8ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.845852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bd5d69f6-73f5-4aa5-a847-2975f743c8ea None None] MFA Rules not processed for user `184bf14f33464130ab62ed1853bf4f8d`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:01:21.847038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bd5d69f6-73f5-4aa5-a847-2975f743c8ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:01:21.891557 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:01:21.899262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 750/1504] 10.222.0.25 () {58 vars in 1060 bytes} [Fri Jun 10 13:01:21 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:01:21.906865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b48654b-8283-452d-a0b9-c131a2162b4c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:21.917874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b48654b-8283-452d-a0b9-c131a2162b4c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:21.921347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4141cfed-9998-4e0f-b3e7-264fe4cc1715 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:21.921977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4141cfed-9998-4e0f-b3e7-264fe4cc1715 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.922283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4141cfed-9998-4e0f-b3e7-264fe4cc1715 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.922560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4141cfed-9998-4e0f-b3e7-264fe4cc1715 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.930630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4141cfed-9998-4e0f-b3e7-264fe4cc1715 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:21.932064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4141cfed-9998-4e0f-b3e7-264fe4cc1715 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:21.940586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 755/1505] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:01:21 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:01:21.983834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48509424-665a-402e-a1e3-2caef7b40956 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:21.984122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48509424-665a-402e-a1e3-2caef7b40956 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:21.984400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48509424-665a-402e-a1e3-2caef7b40956 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:21.985745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 751/1506] 10.222.0.25 () {52 vars in 829 bytes} [Fri Jun 10 13:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:01:21.989281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4141cfed-9998-4e0f-b3e7-264fe4cc1715 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:22.050928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4141cfed-9998-4e0f-b3e7-264fe4cc1715 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:22.055346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:22.056206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:22.056520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:22.056824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] PATH_INFO: `/v3/users/769c5b2dea984d47b43c9b0749ab8b00` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:22.058002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] RBAC: Authorizing `identity:delete_user(user_id=769c5b2dea984d47b43c9b0749ab8b00)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:22.059020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WARNING py.warnings [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:01:22.059020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: warnings.warn(msg) Jun 10 13:01:22.059020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: Jun 10 13:01:22.060218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:22.299191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '769c5b2dea984d47b43c9b0749ab8b00'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:22.299835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '769c5b2dea984d47b43c9b0749ab8b00'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:22.311780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '769c5b2dea984d47b43c9b0749ab8b00'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:22.412819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '769c5b2dea984d47b43c9b0749ab8b00'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:22.413592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '769c5b2dea984d47b43c9b0749ab8b00'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:22.430291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:01:22.437985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 756/1507] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:01:21 2022] DELETE /identity/v3/users/769c5b2dea984d47b43c9b0749ab8b00 => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:01:22.442431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2761ba-557f-42d2-a318-a818d93b3b4e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:22.442892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2761ba-557f-42d2-a318-a818d93b3b4e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:22.443004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2761ba-557f-42d2-a318-a818d93b3b4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:22.461024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-fa2761ba-557f-42d2-a318-a818d93b3b4e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:01:22.462221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fa2761ba-557f-42d2-a318-a818d93b3b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:01:22.520964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 752/1508] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:01:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:01:22.530909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:22.532280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:01:22.590796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edd7c0e-eaf8-42cf-bf2e-0a0d0cfb6627 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:22.596009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172d4af6-c2a6-49f6-be30-5a02f110d9d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:22.597008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172d4af6-c2a6-49f6-be30-5a02f110d9d2 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:22.597444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172d4af6-c2a6-49f6-be30-5a02f110d9d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:22.597864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172d4af6-c2a6-49f6-be30-5a02f110d9d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:22.611472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172d4af6-c2a6-49f6-be30-5a02f110d9d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:22.613831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172d4af6-c2a6-49f6-be30-5a02f110d9d2 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:22.626398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 757/1509] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:01:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:01:23.122354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2761ba-557f-42d2-a318-a818d93b3b4e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:01:23.123898 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fa2761ba-557f-42d2-a318-a818d93b3b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:01:23.136717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2761ba-557f-42d2-a318-a818d93b3b4e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:01:23.141397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:01:23.142264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:01:23.142543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:01:23.142820 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] PATH_INFO: `/v3/projects/e0cc6fc3cde240e689739878ec42e699` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:01:23.144075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0cc6fc3cde240e689739878ec42e699)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:01:23.145158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING py.warnings [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:01:23.145158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: warnings.warn(msg) Jun 10 13:01:23.145158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: Jun 10 13:01:23.146391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:01:23.271980 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0cc6fc3cde240e689739878ec42e699', 'request_id': 'req-1d024803-990d-4175-9fed-a2002488666b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:23.330770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0cc6fc3cde240e689739878ec42e699', 'request_id': 'req-1d024803-990d-4175-9fed-a2002488666b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:23.345523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0cc6fc3cde240e689739878ec42e699', 'request_id': 'req-1d024803-990d-4175-9fed-a2002488666b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:23.357065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] The token cache is being invalidate because project e0cc6fc3cde240e689739878ec42e699 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:01:23.357358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:23.358111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1d024803-990d-4175-9fed-a2002488666b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:01:23.359650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 753/1510] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:01:23 2022] DELETE /identity/v3/projects/e0cc6fc3cde240e689739878ec42e699 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:01:32.621553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:DELETE=1,keystone:SELECT=28 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:01:33.354003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=55,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:03:01.153243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca45c09-b91b-443b-bbc5-05d7fef1f0ed None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:01.153858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca45c09-b91b-443b-bbc5-05d7fef1f0ed None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:01.153858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca45c09-b91b-443b-bbc5-05d7fef1f0ed None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:01.155165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 758/1511] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 13:03:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:03:01.160337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2934b79e-5ff0-41d4-a8c0-db578ff6b3a6 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:01.160795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2934b79e-5ff0-41d4-a8c0-db578ff6b3a6 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:01.161040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2934b79e-5ff0-41d4-a8c0-db578ff6b3a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:01.178740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-2934b79e-5ff0-41d4-a8c0-db578ff6b3a6 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:01.179955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2934b79e-5ff0-41d4-a8c0-db578ff6b3a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:01.238367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 754/1512] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 13:03:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:01.405925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca45c09-b91b-443b-bbc5-05d7fef1f0ed None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:03:01.407554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0ca45c09-b91b-443b-bbc5-05d7fef1f0ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:01.480029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca45c09-b91b-443b-bbc5-05d7fef1f0ed None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:03:01.485291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d389a5-cb7a-4fcf-9e52-7e14d4fd4716 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:03:01.485926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d389a5-cb7a-4fcf-9e52-7e14d4fd4716 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:01.486211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d389a5-cb7a-4fcf-9e52-7e14d4fd4716 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:01.486488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d389a5-cb7a-4fcf-9e52-7e14d4fd4716 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:01.494385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d389a5-cb7a-4fcf-9e52-7e14d4fd4716 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:03:01.496212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5d389a5-cb7a-4fcf-9e52-7e14d4fd4716 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:03:01.505909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 759/1513] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:03:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:03:02.370899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a267595c-f698-4763-8978-2e7b28d112c4 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:02.371377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a267595c-f698-4763-8978-2e7b28d112c4 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:02.371488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a267595c-f698-4763-8978-2e7b28d112c4 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:02.372840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 755/1514] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 13:03:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:03:02.377696 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be81a76-a92d-4fc2-901d-b974f1411d00 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:02.378140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be81a76-a92d-4fc2-901d-b974f1411d00 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:02.378292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be81a76-a92d-4fc2-901d-b974f1411d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:02.396315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-2be81a76-a92d-4fc2-901d-b974f1411d00 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:02.397608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2be81a76-a92d-4fc2-901d-b974f1411d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:02.405271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 760/1515] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 13:03:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:02.560873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a267595c-f698-4763-8978-2e7b28d112c4 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:03:02.574893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a267595c-f698-4763-8978-2e7b28d112c4 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:03:02.580172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9823b2d0-5fac-4ffb-9024-a63dcb33a60b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:03:02.580954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9823b2d0-5fac-4ffb-9024-a63dcb33a60b service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:02.581241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9823b2d0-5fac-4ffb-9024-a63dcb33a60b service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:02.581525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9823b2d0-5fac-4ffb-9024-a63dcb33a60b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:02.590083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9823b2d0-5fac-4ffb-9024-a63dcb33a60b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:03:02.591658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9823b2d0-5fac-4ffb-9024-a63dcb33a60b service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:03:02.601525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 756/1516] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:03:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:03:03.241013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11091007-1d2c-4abf-9888-d7f5a71f8c9f None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:03.241305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11091007-1d2c-4abf-9888-d7f5a71f8c9f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:03.241586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11091007-1d2c-4abf-9888-d7f5a71f8c9f None None] PATH_INFO: `/v3` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:03.242901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 761/1517] 10.222.0.24 () {54 vars in 925 bytes} [Fri Jun 10 13:03:03 2022] GET /identity/v3 => generated 255 bytes in 4 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:03:03.250106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606256e-7d25-4d37-9ad8-f49c38d79dc5 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:03.250401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606256e-7d25-4d37-9ad8-f49c38d79dc5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:03.250689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e606256e-7d25-4d37-9ad8-f49c38d79dc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:03.269209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e606256e-7d25-4d37-9ad8-f49c38d79dc5 None None] MFA Rules not processed for user `0c315cfb1aa44f3c9a755f3cb8e5cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:03.270397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e606256e-7d25-4d37-9ad8-f49c38d79dc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:03.330731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 757/1518] 10.222.0.24 () {58 vars in 1015 bytes} [Fri Jun 10 13:03:03 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:12.391240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=37 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:03:13.322167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=62 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:03:13.339374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f679dc8-30c6-43e0-a1f0-bafbafdb0872 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:13.339670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f679dc8-30c6-43e0-a1f0-bafbafdb0872 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:13.339951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f679dc8-30c6-43e0-a1f0-bafbafdb0872 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:13.341437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 762/1519] 10.222.0.25 () {54 vars in 960 bytes} [Fri Jun 10 13:03:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:03:13.346705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8312b306-0f7f-474f-9164-6eb70f95e40d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:13.347005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8312b306-0f7f-474f-9164-6eb70f95e40d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:13.347294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8312b306-0f7f-474f-9164-6eb70f95e40d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:13.366178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-8312b306-0f7f-474f-9164-6eb70f95e40d None None] MFA Rules not processed for user `0c315cfb1aa44f3c9a755f3cb8e5cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:13.367389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8312b306-0f7f-474f-9164-6eb70f95e40d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:13.376130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 758/1520] 10.222.0.25 () {58 vars in 1057 bytes} [Fri Jun 10 13:03:13 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:13.383653 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f679dc8-30c6-43e0-a1f0-bafbafdb0872 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:03:13.396706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f679dc8-30c6-43e0-a1f0-bafbafdb0872 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:03:13.401616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182fc365-c831-486d-a6ec-27ca3b9d3a5d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:03:13.402286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182fc365-c831-486d-a6ec-27ca3b9d3a5d service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:13.402572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182fc365-c831-486d-a6ec-27ca3b9d3a5d service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:13.402849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182fc365-c831-486d-a6ec-27ca3b9d3a5d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:13.412242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182fc365-c831-486d-a6ec-27ca3b9d3a5d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:03:13.413870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182fc365-c831-486d-a6ec-27ca3b9d3a5d service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:03:13.423421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 763/1521] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:03:13 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:03:13.898520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db9d99a-9f61-4d1f-b262-073568ba8ea1 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:13.898808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db9d99a-9f61-4d1f-b262-073568ba8ea1 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:13.899088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db9d99a-9f61-4d1f-b262-073568ba8ea1 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:13.900503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 759/1522] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 13:03:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:03:13.905187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b8ecdf-4f57-4719-9ab4-0579c8ce18a2 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:13.905514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b8ecdf-4f57-4719-9ab4-0579c8ce18a2 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:13.905802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b8ecdf-4f57-4719-9ab4-0579c8ce18a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:13.923576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-75b8ecdf-4f57-4719-9ab4-0579c8ce18a2 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:13.924698 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-75b8ecdf-4f57-4719-9ab4-0579c8ce18a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:13.932968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 764/1523] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 13:03:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:14.088458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db9d99a-9f61-4d1f-b262-073568ba8ea1 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:03:14.102757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db9d99a-9f61-4d1f-b262-073568ba8ea1 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:03:14.108128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24dc63f-51a1-47c2-9920-c5413b42b6f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:03:14.108775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24dc63f-51a1-47c2-9920-c5413b42b6f5 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:14.109089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24dc63f-51a1-47c2-9920-c5413b42b6f5 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:14.109373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24dc63f-51a1-47c2-9920-c5413b42b6f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:14.118887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24dc63f-51a1-47c2-9920-c5413b42b6f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:03:14.121493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24dc63f-51a1-47c2-9920-c5413b42b6f5 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:03:14.132168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 760/1524] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:03:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:03:15.016486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690616e2-dc4d-4607-864c-42876820adb2 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:15.017055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690616e2-dc4d-4607-864c-42876820adb2 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:15.017055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-690616e2-dc4d-4607-864c-42876820adb2 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:15.018603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 765/1525] 127.0.0.1 () {54 vars in 901 bytes} [Fri Jun 10 13:03:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:03:15.023875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5830d-b691-4aee-b73b-c56d220edd54 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:15.024288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5830d-b691-4aee-b73b-c56d220edd54 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:15.024431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b5830d-b691-4aee-b73b-c56d220edd54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:15.042238 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-80b5830d-b691-4aee-b73b-c56d220edd54 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:15.043442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-80b5830d-b691-4aee-b73b-c56d220edd54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:15.051731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 761/1526] 127.0.0.1 () {58 vars in 998 bytes} [Fri Jun 10 13:03:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:15.203444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690616e2-dc4d-4607-864c-42876820adb2 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:03:15.218173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-690616e2-dc4d-4607-864c-42876820adb2 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:03:15.224301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd778c89-e009-46c2-9bb9-14b4ad7b48a1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:03:15.224978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd778c89-e009-46c2-9bb9-14b4ad7b48a1 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:15.225271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd778c89-e009-46c2-9bb9-14b4ad7b48a1 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:15.225551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd778c89-e009-46c2-9bb9-14b4ad7b48a1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:15.233633 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd778c89-e009-46c2-9bb9-14b4ad7b48a1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:03:15.235230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd778c89-e009-46c2-9bb9-14b4ad7b48a1 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:03:15.244714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 766/1527] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:03:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:03:19.312418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df659bbb-dff8-4e01-ab44-4ffdb81bee5f None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:19.312982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df659bbb-dff8-4e01-ab44-4ffdb81bee5f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:19.312982 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df659bbb-dff8-4e01-ab44-4ffdb81bee5f None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:19.314377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 762/1528] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 13:03:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:03:19.319574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313ae88-c5d8-4d1b-a27c-f8778fa9b5eb None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:19.319988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313ae88-c5d8-4d1b-a27c-f8778fa9b5eb None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:19.320138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4313ae88-c5d8-4d1b-a27c-f8778fa9b5eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:19.338117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4313ae88-c5d8-4d1b-a27c-f8778fa9b5eb None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:19.339301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4313ae88-c5d8-4d1b-a27c-f8778fa9b5eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:19.347072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 767/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 13:03:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:19.385617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df659bbb-dff8-4e01-ab44-4ffdb81bee5f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:03:19.387294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-df659bbb-dff8-4e01-ab44-4ffdb81bee5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:19.443843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df659bbb-dff8-4e01-ab44-4ffdb81bee5f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:03:19.447308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-155cbf1d-a7da-44dd-9e80-348e3ec3c083 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:03:19.447935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155cbf1d-a7da-44dd-9e80-348e3ec3c083 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:19.448216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155cbf1d-a7da-44dd-9e80-348e3ec3c083 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:19.448489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155cbf1d-a7da-44dd-9e80-348e3ec3c083 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:19.457078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155cbf1d-a7da-44dd-9e80-348e3ec3c083 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:03:19.458498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-155cbf1d-a7da-44dd-9e80-348e3ec3c083 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:03:19.467571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 763/1530] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:03:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:03:21.556543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a802a1d6-6cbe-41fd-a3f0-2321d9389743 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:21.558725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a802a1d6-6cbe-41fd-a3f0-2321d9389743 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:21.558725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a802a1d6-6cbe-41fd-a3f0-2321d9389743 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:21.558725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 768/1531] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 13:03:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:03:21.563697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a23e96e-8c4a-4d57-b0a6-972206ed9587 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:21.564083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a23e96e-8c4a-4d57-b0a6-972206ed9587 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:21.564254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a23e96e-8c4a-4d57-b0a6-972206ed9587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:21.581408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3a23e96e-8c4a-4d57-b0a6-972206ed9587 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:21.582629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3a23e96e-8c4a-4d57-b0a6-972206ed9587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:21.590346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 764/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 13:03:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:21.626712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a802a1d6-6cbe-41fd-a3f0-2321d9389743 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:03:21.637239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a802a1d6-6cbe-41fd-a3f0-2321d9389743 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:03:21.640667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3049e6-ac17-4fc5-9da0-ce3961dfcf52 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:03:21.641319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3049e6-ac17-4fc5-9da0-ce3961dfcf52 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:21.641600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3049e6-ac17-4fc5-9da0-ce3961dfcf52 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:21.641872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3049e6-ac17-4fc5-9da0-ce3961dfcf52 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:21.650503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3049e6-ac17-4fc5-9da0-ce3961dfcf52 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:03:21.651920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3049e6-ac17-4fc5-9da0-ce3961dfcf52 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:03:21.661562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 769/1533] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:03:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:03:25.323172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8166ca5c-fe83-46d3-8ad8-0ecf99a7886a None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:25.323768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8166ca5c-fe83-46d3-8ad8-0ecf99a7886a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:25.323768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8166ca5c-fe83-46d3-8ad8-0ecf99a7886a None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:25.325126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 765/1534] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 13:03:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:03:25.330177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b32798-9dec-4970-b655-6f6d476af431 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:25.330583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b32798-9dec-4970-b655-6f6d476af431 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:25.330728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34b32798-9dec-4970-b655-6f6d476af431 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:25.351038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-34b32798-9dec-4970-b655-6f6d476af431 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:25.352224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-34b32798-9dec-4970-b655-6f6d476af431 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:25.360823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 770/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 13:03:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:25.624665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8166ca5c-fe83-46d3-8ad8-0ecf99a7886a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:03:25.626449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8166ca5c-fe83-46d3-8ad8-0ecf99a7886a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:25.694549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8166ca5c-fe83-46d3-8ad8-0ecf99a7886a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:03:25.698034 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f7a285-a15c-4eba-a0d2-ffc4a50de431 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:03:25.698674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f7a285-a15c-4eba-a0d2-ffc4a50de431 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:25.698956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f7a285-a15c-4eba-a0d2-ffc4a50de431 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:25.699238 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f7a285-a15c-4eba-a0d2-ffc4a50de431 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:25.708021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f7a285-a15c-4eba-a0d2-ffc4a50de431 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:03:25.709477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1f7a285-a15c-4eba-a0d2-ffc4a50de431 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:03:25.718603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 766/1536] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:03:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:03:27.854411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892f2e4-afc1-49d8-b20c-d32605935680 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:27.854976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892f2e4-afc1-49d8-b20c-d32605935680 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:27.855090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7892f2e4-afc1-49d8-b20c-d32605935680 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:27.856431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 771/1537] 127.0.0.1 () {54 vars in 916 bytes} [Fri Jun 10 13:03:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:03:27.861784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c38c3-2d29-4955-b6b3-6945da8c461b None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:27.862207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c38c3-2d29-4955-b6b3-6945da8c461b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:27.862348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a6c38c3-2d29-4955-b6b3-6945da8c461b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:27.881812 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9a6c38c3-2d29-4955-b6b3-6945da8c461b None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:03:27.883015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9a6c38c3-2d29-4955-b6b3-6945da8c461b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:27.890770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 767/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Fri Jun 10 13:03:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:03:28.150820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7892f2e4-afc1-49d8-b20c-d32605935680 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:03:28.152492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7892f2e4-afc1-49d8-b20c-d32605935680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:03:28.165810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7892f2e4-afc1-49d8-b20c-d32605935680 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:03:28.169288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602f8fb-dba6-4d43-8ea7-21c20577ef12 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:03:28.169916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602f8fb-dba6-4d43-8ea7-21c20577ef12 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:03:28.170199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602f8fb-dba6-4d43-8ea7-21c20577ef12 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:03:28.170474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602f8fb-dba6-4d43-8ea7-21c20577ef12 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:03:28.179698 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9602f8fb-dba6-4d43-8ea7-21c20577ef12 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:03:28.181161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9602f8fb-dba6-4d43-8ea7-21c20577ef12 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:03:28.190362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 772/1539] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:03:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:03:37.876325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=70 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:03:38.186254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=33 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:56.733591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62269db7-c415-4e23-9817-cd803582dbf8 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.742850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62269db7-c415-4e23-9817-cd803582dbf8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.742850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62269db7-c415-4e23-9817-cd803582dbf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.743080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b094ba-36d1-465b-bd4a-c098d314cbdf None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.743080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b094ba-36d1-465b-bd4a-c098d314cbdf None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.743080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b094ba-36d1-465b-bd4a-c098d314cbdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.762791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-62269db7-c415-4e23-9817-cd803582dbf8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:56.763911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-62269db7-c415-4e23-9817-cd803582dbf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:56.768194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-71b094ba-36d1-465b-bd4a-c098d314cbdf None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:56.775599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-71b094ba-36d1-465b-bd4a-c098d314cbdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:56.782038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 768/1540] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:56.786385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10499e59-7d66-4eb6-99c9-28a571a9be4f None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.786385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10499e59-7d66-4eb6-99c9-28a571a9be4f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.786385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10499e59-7d66-4eb6-99c9-28a571a9be4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.788405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 773/1541] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:04:56.799809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e664948b-68cf-4ace-8840-fa109b26c17e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.800466 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e664948b-68cf-4ace-8840-fa109b26c17e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.800992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e664948b-68cf-4ace-8840-fa109b26c17e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.815409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-10499e59-7d66-4eb6-99c9-28a571a9be4f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:56.816725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-10499e59-7d66-4eb6-99c9-28a571a9be4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:56.830228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 769/1542] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:56.831593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ced2df8-a109-47f8-9b09-50ad5595dab8 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.831946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ced2df8-a109-47f8-9b09-50ad5595dab8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.832229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ced2df8-a109-47f8-9b09-50ad5595dab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.839968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e664948b-68cf-4ace-8840-fa109b26c17e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:56.842087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e664948b-68cf-4ace-8840-fa109b26c17e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:56.855989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 774/1543] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:04:56.857699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb0b639-d5cd-403d-b65a-37432b838827 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.860978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb0b639-d5cd-403d-b65a-37432b838827 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.860978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb0b639-d5cd-403d-b65a-37432b838827 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.863823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-7ced2df8-a109-47f8-9b09-50ad5595dab8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:56.865504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7ced2df8-a109-47f8-9b09-50ad5595dab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:56.876635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 770/1544] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:56.877452 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db60b46-6a8d-40a0-8593-fdf4569b7dec None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.877918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db60b46-6a8d-40a0-8593-fdf4569b7dec None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.878311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db60b46-6a8d-40a0-8593-fdf4569b7dec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.889949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5fb0b639-d5cd-403d-b65a-37432b838827 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:56.891155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5fb0b639-d5cd-403d-b65a-37432b838827 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:56.900118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 775/1545] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:56.900299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-0db60b46-6a8d-40a0-8593-fdf4569b7dec None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:56.901243 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0db60b46-6a8d-40a0-8593-fdf4569b7dec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:56.901731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77405186-6e19-4c57-bc75-59aed334e234 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.902413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77405186-6e19-4c57-bc75-59aed334e234 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.902914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77405186-6e19-4c57-bc75-59aed334e234 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.912436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 771/1546] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:56.913940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd136f-9ebb-408e-8026-8a932e6312c1 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.914267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd136f-9ebb-408e-8026-8a932e6312c1 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.914620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febd136f-9ebb-408e-8026-8a932e6312c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.925805 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-77405186-6e19-4c57-bc75-59aed334e234 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:56.927004 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-77405186-6e19-4c57-bc75-59aed334e234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:56.935473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 776/1547] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:56.937313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77405186-6e19-4c57-bc75-59aed334e234 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:56.937513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-febd136f-9ebb-408e-8026-8a932e6312c1 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:56.938814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-febd136f-9ebb-408e-8026-8a932e6312c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:56.947522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 772/1548] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:56.947911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77405186-6e19-4c57-bc75-59aed334e234 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:56.949634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febd136f-9ebb-408e-8026-8a932e6312c1 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:56.959425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e89851f4-9efc-4213-80f9-78142effa125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:56.960127 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89851f4-9efc-4213-80f9-78142effa125 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.960407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89851f4-9efc-4213-80f9-78142effa125 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.960680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e89851f4-9efc-4213-80f9-78142effa125 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.962550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89851f4-9efc-4213-80f9-78142effa125 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:04:56.962550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89851f4-9efc-4213-80f9-78142effa125 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:56.962746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WARNING py.warnings [None req-e89851f4-9efc-4213-80f9-78142effa125 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:04:56.962746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: warnings.warn(msg) Jun 10 13:04:56.962746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: Jun 10 13:04:56.963661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e89851f4-9efc-4213-80f9-78142effa125 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:56.969838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febd136f-9ebb-408e-8026-8a932e6312c1 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:56.976980 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a61fe44-7f9a-4345-a5e5-8b8253536883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:56.981068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 777/1549] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:04:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:04:56.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a61fe44-7f9a-4345-a5e5-8b8253536883 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a61fe44-7f9a-4345-a5e5-8b8253536883 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.981192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a61fe44-7f9a-4345-a5e5-8b8253536883 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.981449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b7230-f548-4089-87f0-c490c9ab2469 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:56.981449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b7230-f548-4089-87f0-c490c9ab2469 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:56.981449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962b7230-f548-4089-87f0-c490c9ab2469 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:56.981703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a61fe44-7f9a-4345-a5e5-8b8253536883 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:04:56.981863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a61fe44-7f9a-4345-a5e5-8b8253536883 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:56.984106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a61fe44-7f9a-4345-a5e5-8b8253536883 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.006266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 773/1550] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:04:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:04:57.006266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759acd6f-80aa-45a4-a3b1-9d4f42368d4e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.006266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759acd6f-80aa-45a4-a3b1-9d4f42368d4e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.006266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759acd6f-80aa-45a4-a3b1-9d4f42368d4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.006771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-962b7230-f548-4089-87f0-c490c9ab2469 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:57.007844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-962b7230-f548-4089-87f0-c490c9ab2469 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:57.020090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 778/1551] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:57.020261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-759acd6f-80aa-45a4-a3b1-9d4f42368d4e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:57.021429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-759acd6f-80aa-45a4-a3b1-9d4f42368d4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:57.021952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65546007-6ae6-4ac2-984a-09450270b152 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.022416 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65546007-6ae6-4ac2-984a-09450270b152 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.022800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65546007-6ae6-4ac2-984a-09450270b152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.030681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 774/1552] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:57.031500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759acd6f-80aa-45a4-a3b1-9d4f42368d4e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.041559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759acd6f-80aa-45a4-a3b1-9d4f42368d4e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.046049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-65546007-6ae6-4ac2-984a-09450270b152 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:57.046635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd420d7-169b-4b11-b0c5-660e05ead83b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.047334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd420d7-169b-4b11-b0c5-660e05ead83b admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.047460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-65546007-6ae6-4ac2-984a-09450270b152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:57.047585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd420d7-169b-4b11-b0c5-660e05ead83b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.047868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd420d7-169b-4b11-b0c5-660e05ead83b admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.048588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd420d7-169b-4b11-b0c5-660e05ead83b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:04:57.048934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd420d7-169b-4b11-b0c5-660e05ead83b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.050549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd420d7-169b-4b11-b0c5-660e05ead83b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.055138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 779/1553] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:57.056429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c315201-a8b6-44ce-8f86-4de5b29f2fa5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.056729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c315201-a8b6-44ce-8f86-4de5b29f2fa5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.057044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c315201-a8b6-44ce-8f86-4de5b29f2fa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.071919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 775/1554] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:04:57.073114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65314955-4513-44ee-b2e6-c53606d29bf7 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.073405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65314955-4513-44ee-b2e6-c53606d29bf7 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.073684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65314955-4513-44ee-b2e6-c53606d29bf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.079713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5c315201-a8b6-44ce-8f86-4de5b29f2fa5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:57.080947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5c315201-a8b6-44ce-8f86-4de5b29f2fa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:57.088818 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 780/1555] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:57.089622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c315201-a8b6-44ce-8f86-4de5b29f2fa5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.091703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-65314955-4513-44ee-b2e6-c53606d29bf7 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:57.093209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-65314955-4513-44ee-b2e6-c53606d29bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:57.099050 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c315201-a8b6-44ce-8f86-4de5b29f2fa5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.102047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 776/1556] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:57.103098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65314955-4513-44ee-b2e6-c53606d29bf7 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.103952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7171fc19-753c-4be1-9fc3-62f4d07f90fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.104607 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7171fc19-753c-4be1-9fc3-62f4d07f90fe admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.105201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7171fc19-753c-4be1-9fc3-62f4d07f90fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.105334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7171fc19-753c-4be1-9fc3-62f4d07f90fe admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.106014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7171fc19-753c-4be1-9fc3-62f4d07f90fe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.108478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7171fc19-753c-4be1-9fc3-62f4d07f90fe admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.113395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65314955-4513-44ee-b2e6-c53606d29bf7 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.118218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d63bcb9-7ae6-47e0-8cb5-6c08ff9bdf1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.118857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d63bcb9-7ae6-47e0-8cb5-6c08ff9bdf1e admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.119143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d63bcb9-7ae6-47e0-8cb5-6c08ff9bdf1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.119424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d63bcb9-7ae6-47e0-8cb5-6c08ff9bdf1e admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.120107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d63bcb9-7ae6-47e0-8cb5-6c08ff9bdf1e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.121155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING py.warnings [None req-6d63bcb9-7ae6-47e0-8cb5-6c08ff9bdf1e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:04:57.121155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: warnings.warn(msg) Jun 10 13:04:57.121155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: Jun 10 13:04:57.122431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d63bcb9-7ae6-47e0-8cb5-6c08ff9bdf1e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.198424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 781/1557] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/projects => generated 405 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:57.199856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dae9511-4b13-44c6-9eea-22ad4c7e3c70 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.200163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dae9511-4b13-44c6-9eea-22ad4c7e3c70 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.200439 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dae9511-4b13-44c6-9eea-22ad4c7e3c70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.217519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-2dae9511-4b13-44c6-9eea-22ad4c7e3c70 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:57.218715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2dae9511-4b13-44c6-9eea-22ad4c7e3c70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:57.228635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 777/1558] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/projects => generated 403 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:57.228635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d63bcb9-7ae6-47e0-8cb5-6c08ff9bdf1e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.280456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 782/1559] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:57.280456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5022d6-8d3d-4c00-b500-1bbb9e8ca5c3 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.280456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5022d6-8d3d-4c00-b500-1bbb9e8ca5c3 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.280456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5022d6-8d3d-4c00-b500-1bbb9e8ca5c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.290855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d63bcb9-7ae6-47e0-8cb5-6c08ff9bdf1e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.297964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-cd5022d6-8d3d-4c00-b500-1bbb9e8ca5c3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:57.299149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-cd5022d6-8d3d-4c00-b500-1bbb9e8ca5c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:57.301500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af53469-2ba2-4aa4-8fba-52bfecc23017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.302175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af53469-2ba2-4aa4-8fba-52bfecc23017 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.302460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af53469-2ba2-4aa4-8fba-52bfecc23017 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.302740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af53469-2ba2-4aa4-8fba-52bfecc23017 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.303470 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af53469-2ba2-4aa4-8fba-52bfecc23017 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:04:57.303780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af53469-2ba2-4aa4-8fba-52bfecc23017 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.305509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af53469-2ba2-4aa4-8fba-52bfecc23017 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.307443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 783/1560] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:57.308057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5022d6-8d3d-4c00-b500-1bbb9e8ca5c3 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.321552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5022d6-8d3d-4c00-b500-1bbb9e8ca5c3 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.327301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2388af-b0e6-400e-9733-cbc8377a3174 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.327474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 778/1561] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:04:57.328213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af53469-2ba2-4aa4-8fba-52bfecc23017 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.328539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2388af-b0e6-400e-9733-cbc8377a3174 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.328859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2388af-b0e6-400e-9733-cbc8377a3174 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.329138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a2388af-b0e6-400e-9733-cbc8377a3174 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.329832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2388af-b0e6-400e-9733-cbc8377a3174 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.334252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a2388af-b0e6-400e-9733-cbc8377a3174 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.338856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af53469-2ba2-4aa4-8fba-52bfecc23017 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.344028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7a7385-282e-4385-bea8-8045fb2eb2e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.344687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7a7385-282e-4385-bea8-8045fb2eb2e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.345003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7a7385-282e-4385-bea8-8045fb2eb2e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.345282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7a7385-282e-4385-bea8-8045fb2eb2e4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.346005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7a7385-282e-4385-bea8-8045fb2eb2e4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:04:57.346314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7a7385-282e-4385-bea8-8045fb2eb2e4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.347937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7a7385-282e-4385-bea8-8045fb2eb2e4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.360044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 779/1562] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:04:57.360895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7a7385-282e-4385-bea8-8045fb2eb2e4 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.370639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7a7385-282e-4385-bea8-8045fb2eb2e4 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.375616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13580dd-05fb-474c-8fed-4d395ac076f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.376235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13580dd-05fb-474c-8fed-4d395ac076f3 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.376520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13580dd-05fb-474c-8fed-4d395ac076f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.376822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13580dd-05fb-474c-8fed-4d395ac076f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.377529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13580dd-05fb-474c-8fed-4d395ac076f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:04:57.377832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13580dd-05fb-474c-8fed-4d395ac076f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.379444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13580dd-05fb-474c-8fed-4d395ac076f3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.392278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 780/1563] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:04:57.393062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13580dd-05fb-474c-8fed-4d395ac076f3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.402576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13580dd-05fb-474c-8fed-4d395ac076f3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.407019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0fc8e3-3872-4358-85e0-dba321c3419a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.407657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0fc8e3-3872-4358-85e0-dba321c3419a admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.407936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0fc8e3-3872-4358-85e0-dba321c3419a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.408209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0fc8e3-3872-4358-85e0-dba321c3419a admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.408993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0fc8e3-3872-4358-85e0-dba321c3419a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.410907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0fc8e3-3872-4358-85e0-dba321c3419a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.449775 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 784/1564] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/projects => generated 363 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:57.451135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2388af-b0e6-400e-9733-cbc8377a3174 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.506930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:57.518417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a2388af-b0e6-400e-9733-cbc8377a3174 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.523060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fd6984-c305-4f89-b9c6-a6f303bb821d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.523741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fd6984-c305-4f89-b9c6-a6f303bb821d admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.524012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fd6984-c305-4f89-b9c6-a6f303bb821d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.524287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6fd6984-c305-4f89-b9c6-a6f303bb821d admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.525005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fd6984-c305-4f89-b9c6-a6f303bb821d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.526027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WARNING py.warnings [None req-b6fd6984-c305-4f89-b9c6-a6f303bb821d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:04:57.526027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: warnings.warn(msg) Jun 10 13:04:57.526027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: Jun 10 13:04:57.527233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6fd6984-c305-4f89-b9c6-a6f303bb821d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.540521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 781/1565] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/users => generated 303 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:57.541193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0fc8e3-3872-4358-85e0-dba321c3419a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.552156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0fc8e3-3872-4358-85e0-dba321c3419a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.556601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438b6c91-e200-4323-a19a-d3f92c0ee0ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.557248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438b6c91-e200-4323-a19a-d3f92c0ee0ef admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.557528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438b6c91-e200-4323-a19a-d3f92c0ee0ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.557799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438b6c91-e200-4323-a19a-d3f92c0ee0ef admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.558501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438b6c91-e200-4323-a19a-d3f92c0ee0ef admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:04:57.558803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438b6c91-e200-4323-a19a-d3f92c0ee0ef admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.560978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438b6c91-e200-4323-a19a-d3f92c0ee0ef admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.572156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 782/1566] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:04:57.574010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438b6c91-e200-4323-a19a-d3f92c0ee0ef admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.578228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:57.583248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438b6c91-e200-4323-a19a-d3f92c0ee0ef admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.587749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e94077c-2813-4ba3-a989-41211db9d517 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.588357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e94077c-2813-4ba3-a989-41211db9d517 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.588629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e94077c-2813-4ba3-a989-41211db9d517 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.588939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e94077c-2813-4ba3-a989-41211db9d517 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.589675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e94077c-2813-4ba3-a989-41211db9d517 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:04:57.589977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e94077c-2813-4ba3-a989-41211db9d517 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.591530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e94077c-2813-4ba3-a989-41211db9d517 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.602947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 783/1567] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:04:57.603683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e94077c-2813-4ba3-a989-41211db9d517 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.614083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e94077c-2813-4ba3-a989-41211db9d517 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.617330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3070182a-b6ec-46f2-8704-feb8a3829206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.617915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3070182a-b6ec-46f2-8704-feb8a3829206 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.618202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3070182a-b6ec-46f2-8704-feb8a3829206 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.618521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3070182a-b6ec-46f2-8704-feb8a3829206 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.619174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3070182a-b6ec-46f2-8704-feb8a3829206 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.621026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3070182a-b6ec-46f2-8704-feb8a3829206 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.643534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 785/1568] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/users => generated 302 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:57.648553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fd6984-c305-4f89-b9c6-a6f303bb821d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.653914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6fd6984-c305-4f89-b9c6-a6f303bb821d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.658827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b213d3a-4a7c-456b-a162-626a2facc8f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.659635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b213d3a-4a7c-456b-a162-626a2facc8f6 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.659922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b213d3a-4a7c-456b-a162-626a2facc8f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.660200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b213d3a-4a7c-456b-a162-626a2facc8f6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.660981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b213d3a-4a7c-456b-a162-626a2facc8f6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.662897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b213d3a-4a7c-456b-a162-626a2facc8f6 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.764828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 784/1569] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/projects => generated 377 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:57.765506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3070182a-b6ec-46f2-8704-feb8a3829206 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.813078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 786/1570] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/projects => generated 355 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:57.813078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b213d3a-4a7c-456b-a162-626a2facc8f6 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.827025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3070182a-b6ec-46f2-8704-feb8a3829206 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.870150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b213d3a-4a7c-456b-a162-626a2facc8f6 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:57.875399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7cd020-8314-49d2-9379-99f11b036266 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.876062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7cd020-8314-49d2-9379-99f11b036266 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.876345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7cd020-8314-49d2-9379-99f11b036266 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.876625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7cd020-8314-49d2-9379-99f11b036266 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.877347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7cd020-8314-49d2-9379-99f11b036266 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.879268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7cd020-8314-49d2-9379-99f11b036266 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.884023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105d401f-917a-42af-a58c-9aaafcfc12aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:57.884709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105d401f-917a-42af-a58c-9aaafcfc12aa admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:57.885013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105d401f-917a-42af-a58c-9aaafcfc12aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:57.885609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-105d401f-917a-42af-a58c-9aaafcfc12aa admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:57.886286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105d401f-917a-42af-a58c-9aaafcfc12aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:57.888322 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-105d401f-917a-42af-a58c-9aaafcfc12aa admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:57.989770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 787/1571] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/users => generated 282 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:57.989770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7cd020-8314-49d2-9379-99f11b036266 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:57.997251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7cd020-8314-49d2-9379-99f11b036266 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.002671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7882ee-f1c2-4ac2-9ebf-48b65151d181 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.003311 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7882ee-f1c2-4ac2-9ebf-48b65151d181 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.003608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7882ee-f1c2-4ac2-9ebf-48b65151d181 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.003869 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7882ee-f1c2-4ac2-9ebf-48b65151d181 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.004581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7882ee-f1c2-4ac2-9ebf-48b65151d181 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.006240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7882ee-f1c2-4ac2-9ebf-48b65151d181 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.011702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 785/1572] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:57 2022] POST /identity/v3/projects => generated 371 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:58.011702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105d401f-917a-42af-a58c-9aaafcfc12aa admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.017144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 788/1573] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:57 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.017813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7882ee-f1c2-4ac2-9ebf-48b65151d181 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.027402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7882ee-f1c2-4ac2-9ebf-48b65151d181 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.077426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-105d401f-917a-42af-a58c-9aaafcfc12aa admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.082080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73658e52-570b-4538-a7a0-557f51cea739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.082732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73658e52-570b-4538-a7a0-557f51cea739 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.083201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73658e52-570b-4538-a7a0-557f51cea739 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.083482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73658e52-570b-4538-a7a0-557f51cea739 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.083592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeefbb9-b563-420a-a012-5eb5a9673d75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.084148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeefbb9-b563-420a-a012-5eb5a9673d75 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.084276 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73658e52-570b-4538-a7a0-557f51cea739 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.084437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeefbb9-b563-420a-a012-5eb5a9673d75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.084718 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeefbb9-b563-420a-a012-5eb5a9673d75 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.085437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeefbb9-b563-420a-a012-5eb5a9673d75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.086113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73658e52-570b-4538-a7a0-557f51cea739 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.087341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeefbb9-b563-420a-a012-5eb5a9673d75 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.089514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:58.186890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 786/1574] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:58 2022] POST /identity/v3/projects => generated 377 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:58.187952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73658e52-570b-4538-a7a0-557f51cea739 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.215610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 789/1575] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:58 2022] POST /identity/v3/projects => generated 389 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:58.217083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeefbb9-b563-420a-a012-5eb5a9673d75 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.241936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:58.251854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73658e52-570b-4538-a7a0-557f51cea739 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.295450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeefbb9-b563-420a-a012-5eb5a9673d75 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.300073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42008d72-ff8f-4f4f-b242-756197e54631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.300741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42008d72-ff8f-4f4f-b242-756197e54631 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.301152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42008d72-ff8f-4f4f-b242-756197e54631 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.301333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42008d72-ff8f-4f4f-b242-756197e54631 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.302030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42008d72-ff8f-4f4f-b242-756197e54631 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.303996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42008d72-ff8f-4f4f-b242-756197e54631 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.310504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb4328d-fdc5-44fb-ba52-dab7a79d7b5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.311166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4328d-fdc5-44fb-ba52-dab7a79d7b5b admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.311502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4328d-fdc5-44fb-ba52-dab7a79d7b5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.311783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4328d-fdc5-44fb-ba52-dab7a79d7b5b admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.312504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb4328d-fdc5-44fb-ba52-dab7a79d7b5b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.314141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfb4328d-fdc5-44fb-ba52-dab7a79d7b5b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.324340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 787/1576] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.325309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb4328d-fdc5-44fb-ba52-dab7a79d7b5b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.335571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb4328d-fdc5-44fb-ba52-dab7a79d7b5b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.340245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e53fc5a-5204-4817-9285-d8ae52205750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.341002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53fc5a-5204-4817-9285-d8ae52205750 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.341939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53fc5a-5204-4817-9285-d8ae52205750 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.342223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53fc5a-5204-4817-9285-d8ae52205750 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.342911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e53fc5a-5204-4817-9285-d8ae52205750 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.344921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e53fc5a-5204-4817-9285-d8ae52205750 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.374588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 790/1577] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:04:58 2022] POST /identity/v3/users => generated 289 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:58.375424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42008d72-ff8f-4f4f-b242-756197e54631 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.384635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42008d72-ff8f-4f4f-b242-756197e54631 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.389199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b4db1e-7d99-487e-bad9-1ac0ff8c9d4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.389900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b4db1e-7d99-487e-bad9-1ac0ff8c9d4a admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.390155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b4db1e-7d99-487e-bad9-1ac0ff8c9d4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.390434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b4db1e-7d99-487e-bad9-1ac0ff8c9d4a admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.391148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b4db1e-7d99-487e-bad9-1ac0ff8c9d4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.392737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b4db1e-7d99-487e-bad9-1ac0ff8c9d4a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.402337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 791/1578] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.403068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b4db1e-7d99-487e-bad9-1ac0ff8c9d4a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.412543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b4db1e-7d99-487e-bad9-1ac0ff8c9d4a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.417068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b563eaa8-8b4e-4c28-b91b-715a995f33c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.417681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b563eaa8-8b4e-4c28-b91b-715a995f33c7 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.417968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b563eaa8-8b4e-4c28-b91b-715a995f33c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.418248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b563eaa8-8b4e-4c28-b91b-715a995f33c7 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.418916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b563eaa8-8b4e-4c28-b91b-715a995f33c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.420624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 788/1579] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:04:58 2022] POST /identity/v3/users => generated 278 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:58.421487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e53fc5a-5204-4817-9285-d8ae52205750 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.421605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b563eaa8-8b4e-4c28-b91b-715a995f33c7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.432261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e53fc5a-5204-4817-9285-d8ae52205750 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.437008 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1475894a-8cea-423f-a003-60909c340f39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.437649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1475894a-8cea-423f-a003-60909c340f39 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.437937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1475894a-8cea-423f-a003-60909c340f39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.438265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1475894a-8cea-423f-a003-60909c340f39 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.438978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1475894a-8cea-423f-a003-60909c340f39 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.440559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1475894a-8cea-423f-a003-60909c340f39 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.451058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 789/1580] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.451803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1475894a-8cea-423f-a003-60909c340f39 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.460935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1475894a-8cea-423f-a003-60909c340f39 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.465297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8604001-3878-40b3-b151-515ecb278e4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.465920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8604001-3878-40b3-b151-515ecb278e4a admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.466718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8604001-3878-40b3-b151-515ecb278e4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.466997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8604001-3878-40b3-b151-515ecb278e4a admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.467646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8604001-3878-40b3-b151-515ecb278e4a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.469522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8604001-3878-40b3-b151-515ecb278e4a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.488994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 792/1581] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:04:58 2022] POST /identity/v3/users => generated 286 bytes in 86 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:58.489992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b563eaa8-8b4e-4c28-b91b-715a995f33c7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.499593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b563eaa8-8b4e-4c28-b91b-715a995f33c7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.504091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad696fd-2ed5-41fc-a952-455ac8ff6375 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.504717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad696fd-2ed5-41fc-a952-455ac8ff6375 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.505033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad696fd-2ed5-41fc-a952-455ac8ff6375 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.505315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad696fd-2ed5-41fc-a952-455ac8ff6375 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.505985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad696fd-2ed5-41fc-a952-455ac8ff6375 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.507973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ad696fd-2ed5-41fc-a952-455ac8ff6375 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.534985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 790/1582] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:04:58 2022] POST /identity/v3/users => generated 289 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:58.535814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8604001-3878-40b3-b151-515ecb278e4a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.545039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8604001-3878-40b3-b151-515ecb278e4a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.549592 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b4ac91-7551-49c5-b81c-8f8aff4be995 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.550224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b4ac91-7551-49c5-b81c-8f8aff4be995 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.550520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b4ac91-7551-49c5-b81c-8f8aff4be995 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.550802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b4ac91-7551-49c5-b81c-8f8aff4be995 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.551515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b4ac91-7551-49c5-b81c-8f8aff4be995 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.553149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b4ac91-7551-49c5-b81c-8f8aff4be995 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.562894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 791/1583] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.563648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b4ac91-7551-49c5-b81c-8f8aff4be995 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.573019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b4ac91-7551-49c5-b81c-8f8aff4be995 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.578068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5828499d-2a5b-4b73-b373-dfd720db9998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.579387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5828499d-2a5b-4b73-b373-dfd720db9998 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.579387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5828499d-2a5b-4b73-b373-dfd720db9998 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.579387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5828499d-2a5b-4b73-b373-dfd720db9998 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.579892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5828499d-2a5b-4b73-b373-dfd720db9998 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.581490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5828499d-2a5b-4b73-b373-dfd720db9998 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.594877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 792/1584] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.594877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5828499d-2a5b-4b73-b373-dfd720db9998 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.599191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 793/1585] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:04:58 2022] POST /identity/v3/users => generated 295 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:58.600193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad696fd-2ed5-41fc-a952-455ac8ff6375 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.604621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5828499d-2a5b-4b73-b373-dfd720db9998 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.609738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981eafaa-55c8-4b6e-aae2-ef39f14fa60e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.610407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981eafaa-55c8-4b6e-aae2-ef39f14fa60e admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.610527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad696fd-2ed5-41fc-a952-455ac8ff6375 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.610699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981eafaa-55c8-4b6e-aae2-ef39f14fa60e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.610981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981eafaa-55c8-4b6e-aae2-ef39f14fa60e admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.611752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981eafaa-55c8-4b6e-aae2-ef39f14fa60e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.613390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981eafaa-55c8-4b6e-aae2-ef39f14fa60e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.615123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e800e08c-8951-4f55-8b09-ca4fd62fbae3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.616100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e800e08c-8951-4f55-8b09-ca4fd62fbae3 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.616349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e800e08c-8951-4f55-8b09-ca4fd62fbae3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.616623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e800e08c-8951-4f55-8b09-ca4fd62fbae3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.617373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e800e08c-8951-4f55-8b09-ca4fd62fbae3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.619053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e800e08c-8951-4f55-8b09-ca4fd62fbae3 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.625125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 793/1586] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.625716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981eafaa-55c8-4b6e-aae2-ef39f14fa60e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.630860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 794/1587] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.630860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e800e08c-8951-4f55-8b09-ca4fd62fbae3 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.637578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981eafaa-55c8-4b6e-aae2-ef39f14fa60e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.640298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e800e08c-8951-4f55-8b09-ca4fd62fbae3 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.643404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e0f5b1-c707-4ec4-a2b3-368aa05505cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.644235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e0f5b1-c707-4ec4-a2b3-368aa05505cb admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.644528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e0f5b1-c707-4ec4-a2b3-368aa05505cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.644846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e0f5b1-c707-4ec4-a2b3-368aa05505cb admin admin] PATH_INFO: `/v3/projects/2952f85640f14fbba1e3d7a3f784c521/users/56b679e9ccf94158b47bb3f32c30e322/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.645506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290949ba-8835-4083-bff8-be563e68ac52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.646216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290949ba-8835-4083-bff8-be563e68ac52 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.646498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290949ba-8835-4083-bff8-be563e68ac52 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.646778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290949ba-8835-4083-bff8-be563e68ac52 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.647493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290949ba-8835-4083-bff8-be563e68ac52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.648903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:58.654343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290949ba-8835-4083-bff8-be563e68ac52 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.659881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 795/1588] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.660539 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e0f5b1-c707-4ec4-a2b3-368aa05505cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=2952f85640f14fbba1e3d7a3f784c521, user_id=56b679e9ccf94158b47bb3f32c30e322, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.660728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290949ba-8835-4083-bff8-be563e68ac52 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.661579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING py.warnings [None req-e7e0f5b1-c707-4ec4-a2b3-368aa05505cb admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:04:58.661579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: warnings.warn(msg) Jun 10 13:04:58.661579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: Jun 10 13:04:58.663606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7e0f5b1-c707-4ec4-a2b3-368aa05505cb admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.670950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290949ba-8835-4083-bff8-be563e68ac52 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.675612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db61a3f2-9ca6-4722-b36d-5aa1931dad1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.676254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db61a3f2-9ca6-4722-b36d-5aa1931dad1c admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.676540 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db61a3f2-9ca6-4722-b36d-5aa1931dad1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.676850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db61a3f2-9ca6-4722-b36d-5aa1931dad1c admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.677592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db61a3f2-9ca6-4722-b36d-5aa1931dad1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.679174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db61a3f2-9ca6-4722-b36d-5aa1931dad1c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.692280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 796/1589] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.692280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db61a3f2-9ca6-4722-b36d-5aa1931dad1c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.699234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db61a3f2-9ca6-4722-b36d-5aa1931dad1c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.704165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaffcf4-d7a7-4f6e-ab2f-ccf9bc726789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.705028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaffcf4-d7a7-4f6e-ab2f-ccf9bc726789 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.705716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaffcf4-d7a7-4f6e-ab2f-ccf9bc726789 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.705716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaffcf4-d7a7-4f6e-ab2f-ccf9bc726789 admin admin] PATH_INFO: `/v3/projects/340a9de2a2fb41018431aab744493b31/users/fbe30e8af9324b0e869fc6cf5f155728/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.720020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaffcf4-d7a7-4f6e-ab2f-ccf9bc726789 admin admin] RBAC: Authorizing `identity:create_grant(project_id=340a9de2a2fb41018431aab744493b31, user_id=fbe30e8af9324b0e869fc6cf5f155728, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.722728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaffcf4-d7a7-4f6e-ab2f-ccf9bc726789 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.727760 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 794/1590] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:04:58 2022] PUT /identity/v3/projects/2952f85640f14fbba1e3d7a3f784c521/users/56b679e9ccf94158b47bb3f32c30e322/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:04:58.728608 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e0f5b1-c707-4ec4-a2b3-368aa05505cb admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.788619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e0f5b1-c707-4ec4-a2b3-368aa05505cb admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.793691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99bb866-3104-44f8-8f93-9748e91b09de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.794355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99bb866-3104-44f8-8f93-9748e91b09de admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.794649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99bb866-3104-44f8-8f93-9748e91b09de admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.794934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99bb866-3104-44f8-8f93-9748e91b09de admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.795662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99bb866-3104-44f8-8f93-9748e91b09de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.797326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99bb866-3104-44f8-8f93-9748e91b09de admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.803405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 797/1591] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:04:58 2022] PUT /identity/v3/projects/340a9de2a2fb41018431aab744493b31/users/fbe30e8af9324b0e869fc6cf5f155728/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:04:58.805868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaffcf4-d7a7-4f6e-ab2f-ccf9bc726789 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.809961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 795/1592] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.809961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99bb866-3104-44f8-8f93-9748e91b09de admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.819658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99bb866-3104-44f8-8f93-9748e91b09de admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.867184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaffcf4-d7a7-4f6e-ab2f-ccf9bc726789 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.875107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc37b46-9c8f-4bcc-8303-ebebe5322f84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.875784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc37b46-9c8f-4bcc-8303-ebebe5322f84 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.876075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc37b46-9c8f-4bcc-8303-ebebe5322f84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.876359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc37b46-9c8f-4bcc-8303-ebebe5322f84 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.877124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc37b46-9c8f-4bcc-8303-ebebe5322f84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.878738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc37b46-9c8f-4bcc-8303-ebebe5322f84 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.884966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4136f319-6917-4b60-87e7-38fecb3825ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.885901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4136f319-6917-4b60-87e7-38fecb3825ec admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.886176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4136f319-6917-4b60-87e7-38fecb3825ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.886447 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4136f319-6917-4b60-87e7-38fecb3825ec admin admin] PATH_INFO: `/v3/projects/77208dd471cf4142b529ce6c6cb7f618/users/c1cd93116d3e46d998f49cace4fdab17/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.890240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 798/1593] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.890918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc37b46-9c8f-4bcc-8303-ebebe5322f84 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.896858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:58.900676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc37b46-9c8f-4bcc-8303-ebebe5322f84 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.901917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4136f319-6917-4b60-87e7-38fecb3825ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=77208dd471cf4142b529ce6c6cb7f618, user_id=c1cd93116d3e46d998f49cace4fdab17, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.904734 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4136f319-6917-4b60-87e7-38fecb3825ec admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.905730 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0a2b12-3584-4bb1-9cb1-808566c41b09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.906484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0a2b12-3584-4bb1-9cb1-808566c41b09 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.906833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0a2b12-3584-4bb1-9cb1-808566c41b09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.907180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0a2b12-3584-4bb1-9cb1-808566c41b09 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.907900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0a2b12-3584-4bb1-9cb1-808566c41b09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.909578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0a2b12-3584-4bb1-9cb1-808566c41b09 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.920170 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 799/1594] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.920976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0a2b12-3584-4bb1-9cb1-808566c41b09 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.930276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0a2b12-3584-4bb1-9cb1-808566c41b09 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.935916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51dd811-e740-403f-9728-847019505dfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.936520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51dd811-e740-403f-9728-847019505dfa admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.936832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51dd811-e740-403f-9728-847019505dfa admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.937111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51dd811-e740-403f-9728-847019505dfa admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.937801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51dd811-e740-403f-9728-847019505dfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.939366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51dd811-e740-403f-9728-847019505dfa admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.949489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 800/1595] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.950260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51dd811-e740-403f-9728-847019505dfa admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:58.959814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51dd811-e740-403f-9728-847019505dfa admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:58.964284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0986226-a51f-4bc6-a6a3-07934c327e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:58.964915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0986226-a51f-4bc6-a6a3-07934c327e71 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.965201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0986226-a51f-4bc6-a6a3-07934c327e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.965483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0986226-a51f-4bc6-a6a3-07934c327e71 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:58.966171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0986226-a51f-4bc6-a6a3-07934c327e71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:58.967746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0986226-a51f-4bc6-a6a3-07934c327e71 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:58.977618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 801/1596] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:58 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:58.978756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a225186-08c5-42d0-a4ff-01ab0646bc78 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:58.979049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a225186-08c5-42d0-a4ff-01ab0646bc78 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:58.979327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a225186-08c5-42d0-a4ff-01ab0646bc78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 796/1597] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:04:58 2022] PUT /identity/v3/projects/77208dd471cf4142b529ce6c6cb7f618/users/c1cd93116d3e46d998f49cace4fdab17/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:04:59.006299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cbfe00-144b-48e4-9d49-26814e8a53cf None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.006299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cbfe00-144b-48e4-9d49-26814e8a53cf None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.006299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cbfe00-144b-48e4-9d49-26814e8a53cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.013680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5a225186-08c5-42d0-a4ff-01ab0646bc78 None None] MFA Rules not processed for user `56b679e9ccf94158b47bb3f32c30e322`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:59.015267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5a225186-08c5-42d0-a4ff-01ab0646bc78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:59.022719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-c2cbfe00-144b-48e4-9d49-26814e8a53cf None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:59.023985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c2cbfe00-144b-48e4-9d49-26814e8a53cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:59.080072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 802/1598] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:58 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:59.083887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a225186-08c5-42d0-a4ff-01ab0646bc78 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.083994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 797/1599] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:59.083994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cbfe00-144b-48e4-9d49-26814e8a53cf None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.090515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:59.092686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a225186-08c5-42d0-a4ff-01ab0646bc78 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.094431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cbfe00-144b-48e4-9d49-26814e8a53cf None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.097347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ffd10d-5c3c-42d9-9463-591e4ac0e2df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.098222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ffd10d-5c3c-42d9-9463-591e4ac0e2df admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.098587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ffd10d-5c3c-42d9-9463-591e4ac0e2df admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.098862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ffd10d-5c3c-42d9-9463-591e4ac0e2df admin admin] PATH_INFO: `/v3/projects/6dace38cecd44f87becbf6dc213851b3/users/9c8b2c27b69a4776ab6b7b7820da02d9/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.099061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1de5b1-7057-416b-99c9-d2def61c3345 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.099717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1de5b1-7057-416b-99c9-d2def61c3345 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.100003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1de5b1-7057-416b-99c9-d2def61c3345 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.100283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1de5b1-7057-416b-99c9-d2def61c3345 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.101045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1de5b1-7057-416b-99c9-d2def61c3345 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.102668 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1de5b1-7057-416b-99c9-d2def61c3345 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.116601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 798/1600] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:04:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:04:59.116601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1de5b1-7057-416b-99c9-d2def61c3345 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.116926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ffd10d-5c3c-42d9-9463-591e4ac0e2df admin admin] RBAC: Authorizing `identity:create_grant(project_id=6dace38cecd44f87becbf6dc213851b3, user_id=9c8b2c27b69a4776ab6b7b7820da02d9, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.117355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ffd10d-5c3c-42d9-9463-591e4ac0e2df admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.125319 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1de5b1-7057-416b-99c9-d2def61c3345 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.129945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b45578-8f78-4f90-9ad6-5dce688c9e9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.130819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b45578-8f78-4f90-9ad6-5dce688c9e9b admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.132388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b45578-8f78-4f90-9ad6-5dce688c9e9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.132388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b45578-8f78-4f90-9ad6-5dce688c9e9b admin admin] PATH_INFO: `/v3/projects/82b81b58d92c4c7e999764af469c5683/users/d6617ab375314a9ab8e8276e81da0767/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.146166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b45578-8f78-4f90-9ad6-5dce688c9e9b admin admin] RBAC: Authorizing `identity:create_grant(project_id=82b81b58d92c4c7e999764af469c5683, user_id=d6617ab375314a9ab8e8276e81da0767, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.148849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b45578-8f78-4f90-9ad6-5dce688c9e9b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.196746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 803/1601] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:04:59 2022] PUT /identity/v3/projects/6dace38cecd44f87becbf6dc213851b3/users/9c8b2c27b69a4776ab6b7b7820da02d9/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:04:59.197588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ffd10d-5c3c-42d9-9463-591e4ac0e2df admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.256811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ffd10d-5c3c-42d9-9463-591e4ac0e2df admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.261597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d9d022-0931-4514-9905-4ad2634d9536 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.262480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d9d022-0931-4514-9905-4ad2634d9536 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.262767 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d9d022-0931-4514-9905-4ad2634d9536 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.263615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d9d022-0931-4514-9905-4ad2634d9536 admin admin] PATH_INFO: `/v3/projects/c40c1b8609d249bfb31e3c9bd5cc0623/users/dad62ca49b6f49c3b6f084accc6bc2f1/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.266295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 799/1602] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:04:59 2022] PUT /identity/v3/projects/82b81b58d92c4c7e999764af469c5683/users/d6617ab375314a9ab8e8276e81da0767/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:04:59.275824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b45578-8f78-4f90-9ad6-5dce688c9e9b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.278234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d9d022-0931-4514-9905-4ad2634d9536 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c40c1b8609d249bfb31e3c9bd5cc0623, user_id=dad62ca49b6f49c3b6f084accc6bc2f1, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.280958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d9d022-0931-4514-9905-4ad2634d9536 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.347925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b45578-8f78-4f90-9ad6-5dce688c9e9b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.352885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a2797-8baa-4145-9cbf-1a6ed4488766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.353806 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a2797-8baa-4145-9cbf-1a6ed4488766 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.354089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a2797-8baa-4145-9cbf-1a6ed4488766 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.354368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a2797-8baa-4145-9cbf-1a6ed4488766 admin admin] PATH_INFO: `/v3/projects/72affe24c05f4972a9a30883d2cbc85f/users/43c31865d2d744d6bb809a7048d2c575/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.358026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 804/1603] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:04:59 2022] PUT /identity/v3/projects/c40c1b8609d249bfb31e3c9bd5cc0623/users/dad62ca49b6f49c3b6f084accc6bc2f1/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:04:59.361669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9443bf-6185-4703-9136-6a341e310474 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.361669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9443bf-6185-4703-9136-6a341e310474 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.361669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9443bf-6185-4703-9136-6a341e310474 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.379026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422a2797-8baa-4145-9cbf-1a6ed4488766 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72affe24c05f4972a9a30883d2cbc85f, user_id=43c31865d2d744d6bb809a7048d2c575, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.381741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422a2797-8baa-4145-9cbf-1a6ed4488766 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.386293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5c9443bf-6185-4703-9136-6a341e310474 None None] MFA Rules not processed for user `c1cd93116d3e46d998f49cace4fdab17`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:59.387515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5c9443bf-6185-4703-9136-6a341e310474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:59.395511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:59.442752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 800/1604] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:04:59 2022] PUT /identity/v3/projects/72affe24c05f4972a9a30883d2cbc85f/users/43c31865d2d744d6bb809a7048d2c575/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:04:59.443676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a2797-8baa-4145-9cbf-1a6ed4488766 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.450151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 805/1605] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:59 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:59.451067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9443bf-6185-4703-9136-6a341e310474 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.505106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a2797-8baa-4145-9cbf-1a6ed4488766 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.511212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9443bf-6185-4703-9136-6a341e310474 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.512467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c58d29-5690-410f-a4ef-6aa07910a532 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.513391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c58d29-5690-410f-a4ef-6aa07910a532 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.513726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c58d29-5690-410f-a4ef-6aa07910a532 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.514005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c58d29-5690-410f-a4ef-6aa07910a532 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.514694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c58d29-5690-410f-a4ef-6aa07910a532 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.516664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c58d29-5690-410f-a4ef-6aa07910a532 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.558306 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9325435d-d393-4fd4-b8bb-7d479ac877ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.558948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9325435d-d393-4fd4-b8bb-7d479ac877ac service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.559229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9325435d-d393-4fd4-b8bb-7d479ac877ac service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.559504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9325435d-d393-4fd4-b8bb-7d479ac877ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.568941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9325435d-d393-4fd4-b8bb-7d479ac877ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.570434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9325435d-d393-4fd4-b8bb-7d479ac877ac service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.580019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 806/1606] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:04:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:04:59.580826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9325435d-d393-4fd4-b8bb-7d479ac877ac service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.581489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 801/1607] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:59 2022] POST /identity/v3/projects => generated 401 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:59.582686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01ca64c-770b-48a7-bdb2-2fea6b3cf002 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.582977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01ca64c-770b-48a7-bdb2-2fea6b3cf002 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.583259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01ca64c-770b-48a7-bdb2-2fea6b3cf002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.593602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9325435d-d393-4fd4-b8bb-7d479ac877ac service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.602394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-d01ca64c-770b-48a7-bdb2-2fea6b3cf002 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:59.603671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d01ca64c-770b-48a7-bdb2-2fea6b3cf002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:59.639158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:59.670372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 802/1608] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:59.671891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c1bc26-85b4-454d-814d-86d0665dba97 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.672183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c1bc26-85b4-454d-814d-86d0665dba97 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.672467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c1bc26-85b4-454d-814d-86d0665dba97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.681918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade89bb5-2a57-4530-b869-8d183068ac30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.682995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade89bb5-2a57-4530-b869-8d183068ac30 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.683121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade89bb5-2a57-4530-b869-8d183068ac30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.683445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade89bb5-2a57-4530-b869-8d183068ac30 admin admin] PATH_INFO: `/v3/projects/28f8244decff453e98e2d372a97a563c/users/b1d1712ea523498f92f258383863b906/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.700935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:04:59.706347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade89bb5-2a57-4530-b869-8d183068ac30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=28f8244decff453e98e2d372a97a563c, user_id=b1d1712ea523498f92f258383863b906, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.709077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade89bb5-2a57-4530-b869-8d183068ac30 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.715909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e4c1bc26-85b4-454d-814d-86d0665dba97 None None] MFA Rules not processed for user `d6617ab375314a9ab8e8276e81da0767`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:59.717738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e4c1bc26-85b4-454d-814d-86d0665dba97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:59.782473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 803/1609] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:59 2022] POST /identity/v3/auth/tokens => generated 3222 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:59.783758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e3e268-bbbd-4c20-9f47-e4317158fc80 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.784045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e3e268-bbbd-4c20-9f47-e4317158fc80 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.784353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e3e268-bbbd-4c20-9f47-e4317158fc80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.795591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 807/1610] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:04:59 2022] PUT /identity/v3/projects/28f8244decff453e98e2d372a97a563c/users/b1d1712ea523498f92f258383863b906/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:04:59.795806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f3152-2d4a-401b-be6d-a043b07054a5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.797304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f3152-2d4a-401b-be6d-a043b07054a5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.797304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df4f3152-2d4a-401b-be6d-a043b07054a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.813474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-df4f3152-2d4a-401b-be6d-a043b07054a5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:59.813724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-30e3e268-bbbd-4c20-9f47-e4317158fc80 None None] MFA Rules not processed for user `dad62ca49b6f49c3b6f084accc6bc2f1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:04:59.814737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-30e3e268-bbbd-4c20-9f47-e4317158fc80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:59.814985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-df4f3152-2d4a-401b-be6d-a043b07054a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:04:59.881776 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 808/1611] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:59.882573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f3152-2d4a-401b-be6d-a043b07054a5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.894721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df4f3152-2d4a-401b-be6d-a043b07054a5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.897911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2217167a-4732-436f-8cae-103e0af2e583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.898810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2217167a-4732-436f-8cae-103e0af2e583 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.899009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2217167a-4732-436f-8cae-103e0af2e583 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.899286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2217167a-4732-436f-8cae-103e0af2e583 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.900032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2217167a-4732-436f-8cae-103e0af2e583 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.902092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2217167a-4732-436f-8cae-103e0af2e583 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.905141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 804/1612] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:04:59 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:04:59.906526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e3e268-bbbd-4c20-9f47-e4317158fc80 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.921354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30e3e268-bbbd-4c20-9f47-e4317158fc80 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:04:59.927855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6a6189-fabc-4310-b06e-509c527fbc76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:04:59.928544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6a6189-fabc-4310-b06e-509c527fbc76 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:04:59.928876 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6a6189-fabc-4310-b06e-509c527fbc76 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:04:59.929157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6a6189-fabc-4310-b06e-509c527fbc76 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:04:59.929957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6a6189-fabc-4310-b06e-509c527fbc76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:04:59.932363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6a6189-fabc-4310-b06e-509c527fbc76 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:04:59.995104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 809/1613] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:04:59 2022] POST /identity/v3/projects => generated 365 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:04:59.995104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2217167a-4732-436f-8cae-103e0af2e583 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:04:59.995104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2217167a-4732-436f-8cae-103e0af2e583 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:00.015551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 805/1614] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:04:59 2022] POST /identity/v3/users => generated 307 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:00.016729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6a6189-fabc-4310-b06e-509c527fbc76 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.027104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6a6189-fabc-4310-b06e-509c527fbc76 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.055383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2217167a-4732-436f-8cae-103e0af2e583 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.059506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5d2397c-f509-41c1-a55a-9f84a95f6c6a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.060173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d2397c-f509-41c1-a55a-9f84a95f6c6a service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.060455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d2397c-f509-41c1-a55a-9f84a95f6c6a service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.060751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5d2397c-f509-41c1-a55a-9f84a95f6c6a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.066401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:00.087212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b3362c-046c-4dca-a785-4cc02a50da25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.087877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b3362c-046c-4dca-a785-4cc02a50da25 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.088211 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b3362c-046c-4dca-a785-4cc02a50da25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.088489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b3362c-046c-4dca-a785-4cc02a50da25 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.089230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b3362c-046c-4dca-a785-4cc02a50da25 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.091705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b3362c-046c-4dca-a785-4cc02a50da25 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.127290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d2397c-f509-41c1-a55a-9f84a95f6c6a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.128727 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5d2397c-f509-41c1-a55a-9f84a95f6c6a service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.139711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 810/1615] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:04:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:00.141138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e74354-a5a3-4305-aba2-6ea670bf76df None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.141429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e74354-a5a3-4305-aba2-6ea670bf76df None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.141706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40e74354-a5a3-4305-aba2-6ea670bf76df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.160172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-40e74354-a5a3-4305-aba2-6ea670bf76df None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:00.161406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-40e74354-a5a3-4305-aba2-6ea670bf76df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:00.169273 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 811/1616] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:00.170154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e74354-a5a3-4305-aba2-6ea670bf76df None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.182747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40e74354-a5a3-4305-aba2-6ea670bf76df None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.186431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e188c8cd-58c0-4707-83ea-5fd99363c991 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.187082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188c8cd-58c0-4707-83ea-5fd99363c991 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.187371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188c8cd-58c0-4707-83ea-5fd99363c991 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.187778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e188c8cd-58c0-4707-83ea-5fd99363c991 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.197197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e188c8cd-58c0-4707-83ea-5fd99363c991 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.198665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e188c8cd-58c0-4707-83ea-5fd99363c991 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.204389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:00.216884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 812/1617] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:00.216884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb71f9e5-6b38-4731-a31b-205306b1cad6 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.216884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb71f9e5-6b38-4731-a31b-205306b1cad6 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.216884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb71f9e5-6b38-4731-a31b-205306b1cad6 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.218078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 813/1619] 10.222.0.25 () {54 vars in 961 bytes} [Fri Jun 10 13:05:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:00.218381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 806/1619] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:00 2022] POST /identity/v3/projects => generated 357 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:00.218668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb71f9e5-6b38-4731-a31b-205306b1cad6 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.220172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b3362c-046c-4dca-a785-4cc02a50da25 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.290322 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b3362c-046c-4dca-a785-4cc02a50da25 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.293771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb71f9e5-6b38-4731-a31b-205306b1cad6 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.295118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f38221-1d34-4a81-ac45-e9191ebaff4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.295781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f38221-1d34-4a81-ac45-e9191ebaff4f admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.296064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f38221-1d34-4a81-ac45-e9191ebaff4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.296341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f38221-1d34-4a81-ac45-e9191ebaff4f admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.297095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f38221-1d34-4a81-ac45-e9191ebaff4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.299042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffc9b41-fd2e-465a-a291-75003068ca67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.299717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffc9b41-fd2e-465a-a291-75003068ca67 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.300003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffc9b41-fd2e-465a-a291-75003068ca67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.300283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bffc9b41-fd2e-465a-a291-75003068ca67 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.301015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffc9b41-fd2e-465a-a291-75003068ca67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.301719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f38221-1d34-4a81-ac45-e9191ebaff4f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.302978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bffc9b41-fd2e-465a-a291-75003068ca67 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.314352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 807/1620] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:00 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:00.315199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f38221-1d34-4a81-ac45-e9191ebaff4f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.385063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 814/1621] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:00 2022] POST /identity/v3/users => generated 289 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:00.386469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffc9b41-fd2e-465a-a291-75003068ca67 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.390591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f38221-1d34-4a81-ac45-e9191ebaff4f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.395030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2131e1-726e-425c-a1c7-4d3fa563f04f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.395774 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2131e1-726e-425c-a1c7-4d3fa563f04f service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.396130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2131e1-726e-425c-a1c7-4d3fa563f04f service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.396544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2131e1-726e-425c-a1c7-4d3fa563f04f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.400082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bffc9b41-fd2e-465a-a291-75003068ca67 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.409385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2131e1-726e-425c-a1c7-4d3fa563f04f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.410212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039b2204-84cb-46ff-85a0-109e59699a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.411100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039b2204-84cb-46ff-85a0-109e59699a8b admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.411225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2131e1-726e-425c-a1c7-4d3fa563f04f service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.411701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039b2204-84cb-46ff-85a0-109e59699a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.412209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039b2204-84cb-46ff-85a0-109e59699a8b admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.413488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039b2204-84cb-46ff-85a0-109e59699a8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.416323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039b2204-84cb-46ff-85a0-109e59699a8b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.425082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 808/1622] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:00.426726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ca4da-7efa-46be-8ff2-964c3f05dcb8 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.427072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ca4da-7efa-46be-8ff2-964c3f05dcb8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.427366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955ca4da-7efa-46be-8ff2-964c3f05dcb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.447127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-955ca4da-7efa-46be-8ff2-964c3f05dcb8 None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:00.448482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-955ca4da-7efa-46be-8ff2-964c3f05dcb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:00.511663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:00.523198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 815/1623] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:00 2022] POST /identity/v3/users => generated 285 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:00.524524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039b2204-84cb-46ff-85a0-109e59699a8b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.532955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 809/1624] 10.222.0.25 () {58 vars in 1058 bytes} [Fri Jun 10 13:05:00 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:00.533919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955ca4da-7efa-46be-8ff2-964c3f05dcb8 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.546311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955ca4da-7efa-46be-8ff2-964c3f05dcb8 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.547803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039b2204-84cb-46ff-85a0-109e59699a8b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.552305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bbc0d1-3c76-4a26-993b-0b81b00824d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.552448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bbc0d1-3c76-4a26-993b-0b81b00824d2 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.552926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bbc0d1-3c76-4a26-993b-0b81b00824d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.553304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bbc0d1-3c76-4a26-993b-0b81b00824d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.553940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa7d773-6abd-4e4e-a158-52dc68a096ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.554127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bbc0d1-3c76-4a26-993b-0b81b00824d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.554940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7d773-6abd-4e4e-a158-52dc68a096ec admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.555325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7d773-6abd-4e4e-a158-52dc68a096ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.555681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7d773-6abd-4e4e-a158-52dc68a096ec admin admin] PATH_INFO: `/v3/projects/724ce3cd23ae46eb934b04eb947f4236/users/147e3838103847fd9d6852c8d43b09b6/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.555901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bbc0d1-3c76-4a26-993b-0b81b00824d2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.572529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 810/1625] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:00 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:00.575392 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bbc0d1-3c76-4a26-993b-0b81b00824d2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.576746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa7d773-6abd-4e4e-a158-52dc68a096ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=724ce3cd23ae46eb934b04eb947f4236, user_id=147e3838103847fd9d6852c8d43b09b6, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.580191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa7d773-6abd-4e4e-a158-52dc68a096ec admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.585583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bbc0d1-3c76-4a26-993b-0b81b00824d2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.591419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245ea4d8-98d8-400c-bf08-7583ec209570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.592150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245ea4d8-98d8-400c-bf08-7583ec209570 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.592922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245ea4d8-98d8-400c-bf08-7583ec209570 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.592922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245ea4d8-98d8-400c-bf08-7583ec209570 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.593582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245ea4d8-98d8-400c-bf08-7583ec209570 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.595215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245ea4d8-98d8-400c-bf08-7583ec209570 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.609504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 811/1626] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:00 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:00.609504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245ea4d8-98d8-400c-bf08-7583ec209570 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.623463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245ea4d8-98d8-400c-bf08-7583ec209570 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.629510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcf4f20-71d8-49f2-9f24-b7452b5160b7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.631589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcf4f20-71d8-49f2-9f24-b7452b5160b7 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.631589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcf4f20-71d8-49f2-9f24-b7452b5160b7 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.631589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dcf4f20-71d8-49f2-9f24-b7452b5160b7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.655075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 816/1627] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:00 2022] PUT /identity/v3/projects/724ce3cd23ae46eb934b04eb947f4236/users/147e3838103847fd9d6852c8d43b09b6/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:00.658250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa7d773-6abd-4e4e-a158-52dc68a096ec admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.744546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcf4f20-71d8-49f2-9f24-b7452b5160b7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.746243 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dcf4f20-71d8-49f2-9f24-b7452b5160b7 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.756253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 812/1628] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:00 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:00.757428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcf4f20-71d8-49f2-9f24-b7452b5160b7 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.769928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dcf4f20-71d8-49f2-9f24-b7452b5160b7 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.775021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa7d773-6abd-4e4e-a158-52dc68a096ec admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:00.776143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4281c60d-dc55-493e-b7a2-24352f155bce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.777322 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281c60d-dc55-493e-b7a2-24352f155bce admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.777781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281c60d-dc55-493e-b7a2-24352f155bce admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.778213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4281c60d-dc55-493e-b7a2-24352f155bce admin admin] PATH_INFO: `/v3/projects/604f78c1c14a4bdba20916e82b772038/users/d138f9e1a64f4b298843c1dc9cf3fba1/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.779459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a5de1-82de-4a43-8762-e5b16dcc23aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:00.780426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a5de1-82de-4a43-8762-e5b16dcc23aa admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:00.780833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a5de1-82de-4a43-8762-e5b16dcc23aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:00.781173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669a5de1-82de-4a43-8762-e5b16dcc23aa admin admin] PATH_INFO: `/v3/projects/cccd79a54d644360890f79e164e18fc3/users/e748601d3d2f48449abfd1bd384c5126/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:00.798138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4281c60d-dc55-493e-b7a2-24352f155bce admin admin] RBAC: Authorizing `identity:create_grant(project_id=604f78c1c14a4bdba20916e82b772038, user_id=d138f9e1a64f4b298843c1dc9cf3fba1, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.801691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4281c60d-dc55-493e-b7a2-24352f155bce admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.815092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669a5de1-82de-4a43-8762-e5b16dcc23aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=cccd79a54d644360890f79e164e18fc3, user_id=e748601d3d2f48449abfd1bd384c5126, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:00.820684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669a5de1-82de-4a43-8762-e5b16dcc23aa admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:00.895289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 813/1629] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:00 2022] PUT /identity/v3/projects/604f78c1c14a4bdba20916e82b772038/users/d138f9e1a64f4b298843c1dc9cf3fba1/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:00.896186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4281c60d-dc55-493e-b7a2-24352f155bce admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:00.958423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 817/1630] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:00 2022] PUT /identity/v3/projects/cccd79a54d644360890f79e164e18fc3/users/e748601d3d2f48449abfd1bd384c5126/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:00.964244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a5de1-82de-4a43-8762-e5b16dcc23aa admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:01.005679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4281c60d-dc55-493e-b7a2-24352f155bce admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:01.077367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:01.100535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:01.114315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f871ccd0-ab2b-4b7f-8c0c-c2c72427ad8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:01.116153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f871ccd0-ab2b-4b7f-8c0c-c2c72427ad8e admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.116704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f871ccd0-ab2b-4b7f-8c0c-c2c72427ad8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.117721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f871ccd0-ab2b-4b7f-8c0c-c2c72427ad8e admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.119678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f871ccd0-ab2b-4b7f-8c0c-c2c72427ad8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:01.126754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f871ccd0-ab2b-4b7f-8c0c-c2c72427ad8e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:01.139154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669a5de1-82de-4a43-8762-e5b16dcc23aa admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:01.144934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 814/1631] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:00 2022] GET /identity/v3/roles => generated 1420 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:01.146036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f871ccd0-ab2b-4b7f-8c0c-c2c72427ad8e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:01.148932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea55bfb2-f1d4-437b-8fdf-1e8fa6c2ef71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:01.151800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea55bfb2-f1d4-437b-8fdf-1e8fa6c2ef71 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.152094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea55bfb2-f1d4-437b-8fdf-1e8fa6c2ef71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.152374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea55bfb2-f1d4-437b-8fdf-1e8fa6c2ef71 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.157596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea55bfb2-f1d4-437b-8fdf-1e8fa6c2ef71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:01.165219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea55bfb2-f1d4-437b-8fdf-1e8fa6c2ef71 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:01.177371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f871ccd0-ab2b-4b7f-8c0c-c2c72427ad8e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:01.190203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 818/1632] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:00 2022] GET /identity/v3/roles => generated 1420 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:01.190203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de2a5e4-dfd5-4451-9b13-f6a91c58a97d None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.190203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de2a5e4-dfd5-4451-9b13-f6a91c58a97d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.197462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de2a5e4-dfd5-4451-9b13-f6a91c58a97d None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.197564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6133491-1df4-4cc8-9b3b-be043654a88a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:01.197564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6133491-1df4-4cc8-9b3b-be043654a88a admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.197564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6133491-1df4-4cc8-9b3b-be043654a88a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.197564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6133491-1df4-4cc8-9b3b-be043654a88a admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.197927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6133491-1df4-4cc8-9b3b-be043654a88a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:01.197927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6133491-1df4-4cc8-9b3b-be043654a88a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:01.199521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 819/1633] 10.222.0.25 () {54 vars in 912 bytes} [Fri Jun 10 13:05:01 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:01.201594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92678309-1350-4657-a4e7-2e3474264ce9 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.202127 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92678309-1350-4657-a4e7-2e3474264ce9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.203698 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92678309-1350-4657-a4e7-2e3474264ce9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.218555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 815/1634] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:01 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:01.219520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6133491-1df4-4cc8-9b3b-be043654a88a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:01.232317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6133491-1df4-4cc8-9b3b-be043654a88a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:01.238612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-92678309-1350-4657-a4e7-2e3474264ce9 None None] MFA Rules not processed for user `fbe30e8af9324b0e869fc6cf5f155728`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:01.239155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8b8bb1-7cb6-48d3-b3ba-3a5355984991 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:01.239848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-92678309-1350-4657-a4e7-2e3474264ce9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:01.240424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8b8bb1-7cb6-48d3-b3ba-3a5355984991 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.240710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8b8bb1-7cb6-48d3-b3ba-3a5355984991 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.241111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8b8bb1-7cb6-48d3-b3ba-3a5355984991 admin admin] PATH_INFO: `/v3/projects/724ce3cd23ae46eb934b04eb947f4236/users/147e3838103847fd9d6852c8d43b09b6/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.244100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8b8bb1-7cb6-48d3-b3ba-3a5355984991 admin admin] RBAC: Authorizing `identity:create_grant(project_id=724ce3cd23ae46eb934b04eb947f4236, user_id=147e3838103847fd9d6852c8d43b09b6, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:01.248172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8b8bb1-7cb6-48d3-b3ba-3a5355984991 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:01.265439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 816/1635] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:01 2022] PUT /identity/v3/projects/724ce3cd23ae46eb934b04eb947f4236/users/147e3838103847fd9d6852c8d43b09b6/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:01.267548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8b8bb1-7cb6-48d3-b3ba-3a5355984991 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:01.352260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 820/1636] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:01 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:01.353940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75628a3c-8e90-4e6e-9eda-5302d156442d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.354394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75628a3c-8e90-4e6e-9eda-5302d156442d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.355274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75628a3c-8e90-4e6e-9eda-5302d156442d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.368717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8b8bb1-7cb6-48d3-b3ba-3a5355984991 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:01.374460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf25147-d80d-4615-a801-1d91c5fa85cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:01.375513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf25147-d80d-4615-a801-1d91c5fa85cb admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.375648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-75628a3c-8e90-4e6e-9eda-5302d156442d None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:01.375933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf25147-d80d-4615-a801-1d91c5fa85cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.376274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf25147-d80d-4615-a801-1d91c5fa85cb admin admin] PATH_INFO: `/v3/projects/604f78c1c14a4bdba20916e82b772038/users/d138f9e1a64f4b298843c1dc9cf3fba1/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.376892 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-75628a3c-8e90-4e6e-9eda-5302d156442d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:01.378733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf25147-d80d-4615-a801-1d91c5fa85cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=604f78c1c14a4bdba20916e82b772038, user_id=d138f9e1a64f4b298843c1dc9cf3fba1, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:01.381511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf25147-d80d-4615-a801-1d91c5fa85cb admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:01.392819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 817/1637] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:01 2022] PUT /identity/v3/projects/604f78c1c14a4bdba20916e82b772038/users/d138f9e1a64f4b298843c1dc9cf3fba1/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:01.393786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf25147-d80d-4615-a801-1d91c5fa85cb admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:01.446676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 821/1638] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:05:01 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:01.448098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c52b508-85d7-4911-9ac3-6f333661a4f1 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.448398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c52b508-85d7-4911-9ac3-6f333661a4f1 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.448683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c52b508-85d7-4911-9ac3-6f333661a4f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.472811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf25147-d80d-4615-a801-1d91c5fa85cb admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:01.476184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28669167-6028-43fa-be2f-6f79096b9ede admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:01.477136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28669167-6028-43fa-be2f-6f79096b9ede admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.477425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28669167-6028-43fa-be2f-6f79096b9ede admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.477710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28669167-6028-43fa-be2f-6f79096b9ede admin admin] PATH_INFO: `/v3/projects/cccd79a54d644360890f79e164e18fc3/users/e748601d3d2f48449abfd1bd384c5126/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.481184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28669167-6028-43fa-be2f-6f79096b9ede admin admin] RBAC: Authorizing `identity:create_grant(project_id=cccd79a54d644360890f79e164e18fc3, user_id=e748601d3d2f48449abfd1bd384c5126, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:01.484279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28669167-6028-43fa-be2f-6f79096b9ede admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:01.488513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-8c52b508-85d7-4911-9ac3-6f333661a4f1 None None] MFA Rules not processed for user `147e3838103847fd9d6852c8d43b09b6`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:01.489797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8c52b508-85d7-4911-9ac3-6f333661a4f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:01.509981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 818/1639] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:01 2022] PUT /identity/v3/projects/cccd79a54d644360890f79e164e18fc3/users/e748601d3d2f48449abfd1bd384c5126/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:01.509981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28669167-6028-43fa-be2f-6f79096b9ede admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:01.582642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 822/1640] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:01 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:01.589009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c035d244-27db-441a-ae53-901f43ce65ab None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.589009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c035d244-27db-441a-ae53-901f43ce65ab None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.589424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c035d244-27db-441a-ae53-901f43ce65ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.603077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28669167-6028-43fa-be2f-6f79096b9ede admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:01.616786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1574f80-64ac-4f80-9bd4-0c7edc02cc9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:01.617764 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1574f80-64ac-4f80-9bd4-0c7edc02cc9e admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.617764 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1574f80-64ac-4f80-9bd4-0c7edc02cc9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.618000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1574f80-64ac-4f80-9bd4-0c7edc02cc9e admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.618706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1574f80-64ac-4f80-9bd4-0c7edc02cc9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:01.620857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1574f80-64ac-4f80-9bd4-0c7edc02cc9e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:01.627089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c035d244-27db-441a-ae53-901f43ce65ab None None] MFA Rules not processed for user `d138f9e1a64f4b298843c1dc9cf3fba1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:01.628520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c035d244-27db-441a-ae53-901f43ce65ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:01.702032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 823/1641] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:01 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:01.702032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c035d244-27db-441a-ae53-901f43ce65ab None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:01.722434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 819/1642] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:01 2022] POST /identity/v3/projects => generated 405 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:01.723932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec23971c-01bc-41e4-9df4-7b3c0aa08b9f None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.724219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec23971c-01bc-41e4-9df4-7b3c0aa08b9f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.724507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec23971c-01bc-41e4-9df4-7b3c0aa08b9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.758694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ec23971c-01bc-41e4-9df4-7b3c0aa08b9f None None] MFA Rules not processed for user `e748601d3d2f48449abfd1bd384c5126`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:01.760943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ec23971c-01bc-41e4-9df4-7b3c0aa08b9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:01.781649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c035d244-27db-441a-ae53-901f43ce65ab None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:01.835624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 820/1643] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:01 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:01.836032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb9f428-0147-4336-bcb0-980785a62508 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:01.836174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec23971c-01bc-41e4-9df4-7b3c0aa08b9f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:01.836255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb9f428-0147-4336-bcb0-980785a62508 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.836255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb9f428-0147-4336-bcb0-980785a62508 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.836255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb9f428-0147-4336-bcb0-980785a62508 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.845970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:01.847764 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec23971c-01bc-41e4-9df4-7b3c0aa08b9f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:01.852265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8240d5-0b16-4d03-8109-a67f5921bb0e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:01.853288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8240d5-0b16-4d03-8109-a67f5921bb0e service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:01.853579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8240d5-0b16-4d03-8109-a67f5921bb0e service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:01.853880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8240d5-0b16-4d03-8109-a67f5921bb0e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:01.921264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb9f428-0147-4336-bcb0-980785a62508 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:01.922930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb9f428-0147-4336-bcb0-980785a62508 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:01.951280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 824/1644] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:01 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:01.955905 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb9f428-0147-4336-bcb0-980785a62508 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:01.964730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8240d5-0b16-4d03-8109-a67f5921bb0e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:01.968354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8240d5-0b16-4d03-8109-a67f5921bb0e service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:01.980044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 821/1645] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:01 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:01.986274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:02.067659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180eefb8-de0e-4d00-9e2e-6111cc0192ca None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.068126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180eefb8-de0e-4d00-9e2e-6111cc0192ca None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.068481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180eefb8-de0e-4d00-9e2e-6111cc0192ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.086438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb9f428-0147-4336-bcb0-980785a62508 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:02.109909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e85657-dc3c-4cf0-89ba-63449d73fc0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:02.112487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e85657-dc3c-4cf0-89ba-63449d73fc0c admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.112487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e85657-dc3c-4cf0-89ba-63449d73fc0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.112487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e85657-dc3c-4cf0-89ba-63449d73fc0c admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.112487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e85657-dc3c-4cf0-89ba-63449d73fc0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:02.114810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e85657-dc3c-4cf0-89ba-63449d73fc0c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:02.116027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-180eefb8-de0e-4d00-9e2e-6111cc0192ca None None] MFA Rules not processed for user `43c31865d2d744d6bb809a7048d2c575`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:02.117365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-180eefb8-de0e-4d00-9e2e-6111cc0192ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:02.203342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 822/1646] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:02 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:02.203913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180eefb8-de0e-4d00-9e2e-6111cc0192ca None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:02.214810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180eefb8-de0e-4d00-9e2e-6111cc0192ca None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:02.218695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d11d1cec-775a-417c-b5d3-5b84374f872c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:02.219417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11d1cec-775a-417c-b5d3-5b84374f872c service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.219709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11d1cec-775a-417c-b5d3-5b84374f872c service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.219990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d11d1cec-775a-417c-b5d3-5b84374f872c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.245724 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 825/1647] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:01 2022] POST /identity/v3/users => generated 309 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:02.247920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51032f07-c600-4004-9b25-6a042558db3c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.250153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51032f07-c600-4004-9b25-6a042558db3c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.250608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51032f07-c600-4004-9b25-6a042558db3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.306143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-51032f07-c600-4004-9b25-6a042558db3c None None] MFA Rules not processed for user `43c31865d2d744d6bb809a7048d2c575`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:02.308467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-51032f07-c600-4004-9b25-6a042558db3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:02.330752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:02.339123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 826/1648] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:02 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:02.340173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51032f07-c600-4004-9b25-6a042558db3c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:02.376413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11d1cec-775a-417c-b5d3-5b84374f872c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:02.376413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d11d1cec-775a-417c-b5d3-5b84374f872c service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:02.376862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51032f07-c600-4004-9b25-6a042558db3c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:02.385343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6d0967-33d2-4f99-a9ea-78b2702894cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:02.386688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6d0967-33d2-4f99-a9ea-78b2702894cf admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.387013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6d0967-33d2-4f99-a9ea-78b2702894cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.387494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6d0967-33d2-4f99-a9ea-78b2702894cf admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.388704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6d0967-33d2-4f99-a9ea-78b2702894cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:02.389974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 823/1649] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:02 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:02.390614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6d0967-33d2-4f99-a9ea-78b2702894cf admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:02.396618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993d7f78-a107-40d2-9a59-dcfbc5d7bd27 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.397082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993d7f78-a107-40d2-9a59-dcfbc5d7bd27 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.397455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993d7f78-a107-40d2-9a59-dcfbc5d7bd27 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.399326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 824/1650] 10.222.0.25 () {54 vars in 961 bytes} [Fri Jun 10 13:05:02 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:02.408940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cb98cc-0775-4e02-a4af-53effe13c37b None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.409477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cb98cc-0775-4e02-a4af-53effe13c37b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.409855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cb98cc-0775-4e02-a4af-53effe13c37b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.415564 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 827/1651] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:02 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:02.420398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6d0967-33d2-4f99-a9ea-78b2702894cf admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:02.433017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6d0967-33d2-4f99-a9ea-78b2702894cf admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:02.437156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-b4cb98cc-0775-4e02-a4af-53effe13c37b None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:02.438678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b4cb98cc-0775-4e02-a4af-53effe13c37b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:02.439498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64849633-5997-4a2e-b431-310cf5480d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:02.440413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64849633-5997-4a2e-b431-310cf5480d9e admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.440705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64849633-5997-4a2e-b431-310cf5480d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.441029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64849633-5997-4a2e-b431-310cf5480d9e admin admin] PATH_INFO: `/v3/projects/1c071c7ca67f4cd788e1fc5b981e6494/users/d2776edb1dc342898968a3f9411caf9f/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.451467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 825/1652] 10.222.0.25 () {58 vars in 1058 bytes} [Fri Jun 10 13:05:02 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:02.458198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64849633-5997-4a2e-b431-310cf5480d9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c071c7ca67f4cd788e1fc5b981e6494, user_id=d2776edb1dc342898968a3f9411caf9f, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:02.459801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cb98cc-0775-4e02-a4af-53effe13c37b None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:02.460976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64849633-5997-4a2e-b431-310cf5480d9e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:02.472872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cb98cc-0775-4e02-a4af-53effe13c37b None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:02.477111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccc1b70-61b4-4f51-9752-8e1489c7cd4c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:02.477778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccc1b70-61b4-4f51-9752-8e1489c7cd4c service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.478061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccc1b70-61b4-4f51-9752-8e1489c7cd4c service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.478339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ccc1b70-61b4-4f51-9752-8e1489c7cd4c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.487148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccc1b70-61b4-4f51-9752-8e1489c7cd4c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:02.488774 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ccc1b70-61b4-4f51-9752-8e1489c7cd4c service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:02.498558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 826/1653] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:02 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:02.538626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccc1b70-61b4-4f51-9752-8e1489c7cd4c service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:02.548685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ccc1b70-61b4-4f51-9752-8e1489c7cd4c service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:02.557616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e466d1-c374-4118-821a-bf95f065db3b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:02.558332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e466d1-c374-4118-821a-bf95f065db3b service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.558618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e466d1-c374-4118-821a-bf95f065db3b service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.558901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1e466d1-c374-4118-821a-bf95f065db3b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.580685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e466d1-c374-4118-821a-bf95f065db3b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:02.583732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1e466d1-c374-4118-821a-bf95f065db3b service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:02.599962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 828/1654] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:02 2022] PUT /identity/v3/projects/1c071c7ca67f4cd788e1fc5b981e6494/users/d2776edb1dc342898968a3f9411caf9f/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:02.604789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64849633-5997-4a2e-b431-310cf5480d9e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:02.614296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 827/1655] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:02 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 76 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:05:02.742299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64849633-5997-4a2e-b431-310cf5480d9e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:02.749072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b9a8ca-5526-484b-8930-ad475abb9c22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:02.749765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b9a8ca-5526-484b-8930-ad475abb9c22 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.750056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b9a8ca-5526-484b-8930-ad475abb9c22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.750342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b9a8ca-5526-484b-8930-ad475abb9c22 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.751264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b9a8ca-5526-484b-8930-ad475abb9c22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:02.752926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b9a8ca-5526-484b-8930-ad475abb9c22 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:02.765366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 829/1656] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:02 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:05:02.769327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e466d1-c374-4118-821a-bf95f065db3b service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:02.791925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1e466d1-c374-4118-821a-bf95f065db3b service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:02.806241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f896d8d1-fce1-4246-9ba1-3024160d6e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:02.807147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f896d8d1-fce1-4246-9ba1-3024160d6e75 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.807421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f896d8d1-fce1-4246-9ba1-3024160d6e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.807703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f896d8d1-fce1-4246-9ba1-3024160d6e75 admin admin] PATH_INFO: `/v3/projects/1c071c7ca67f4cd788e1fc5b981e6494/users/d2776edb1dc342898968a3f9411caf9f/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.811914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f896d8d1-fce1-4246-9ba1-3024160d6e75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c071c7ca67f4cd788e1fc5b981e6494, user_id=d2776edb1dc342898968a3f9411caf9f, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:02.819795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f896d8d1-fce1-4246-9ba1-3024160d6e75 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:02.844400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 828/1657] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:02 2022] PUT /identity/v3/projects/1c071c7ca67f4cd788e1fc5b981e6494/users/d2776edb1dc342898968a3f9411caf9f/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 76 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:02.952161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7881a8-db89-49fa-b981-ad0a452d3fca None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.956304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7881a8-db89-49fa-b981-ad0a452d3fca None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.956304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac7881a8-db89-49fa-b981-ad0a452d3fca None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.956304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 830/1658] 10.222.0.25 () {54 vars in 912 bytes} [Fri Jun 10 13:05:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:02.960442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d3e03d-2b09-455d-b80c-477422367216 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:02.960796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d3e03d-2b09-455d-b80c-477422367216 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:02.961043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d3e03d-2b09-455d-b80c-477422367216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:02.989274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-78d3e03d-2b09-455d-b80c-477422367216 None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:02.990830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-78d3e03d-2b09-455d-b80c-477422367216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:03.126252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 829/1659] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:05:02 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:03.142080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7881a8-db89-49fa-b981-ad0a452d3fca None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:03.144687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ac7881a8-db89-49fa-b981-ad0a452d3fca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:03.166585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d3e03d-2b09-455d-b80c-477422367216 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:03.278805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d3e03d-2b09-455d-b80c-477422367216 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:03.285446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028a66d4-1acf-42a5-ba72-a029d7ebe458 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:03.286598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028a66d4-1acf-42a5-ba72-a029d7ebe458 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:03.286598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028a66d4-1acf-42a5-ba72-a029d7ebe458 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:03.286598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028a66d4-1acf-42a5-ba72-a029d7ebe458 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:03.307183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac7881a8-db89-49fa-b981-ad0a452d3fca None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:03.312328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:03.314324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64cec6d-eedb-48d8-bd66-8cc58faa4980 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a83b4eb626343f3aa5b1c9f3fc828fd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:03.316337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64cec6d-eedb-48d8-bd66-8cc58faa4980 service glance] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:03.317139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64cec6d-eedb-48d8-bd66-8cc58faa4980 service glance] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:03.326684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64cec6d-eedb-48d8-bd66-8cc58faa4980 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:03.362953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64cec6d-eedb-48d8-bd66-8cc58faa4980 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:03.364583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64cec6d-eedb-48d8-bd66-8cc58faa4980 service glance] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:03.391994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028a66d4-1acf-42a5-ba72-a029d7ebe458 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:03.394186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028a66d4-1acf-42a5-ba72-a029d7ebe458 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:03.396265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 831/1660] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:03 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:03.397309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64cec6d-eedb-48d8-bd66-8cc58faa4980 service glance] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:03.416405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64cec6d-eedb-48d8-bd66-8cc58faa4980 service glance] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:03.428677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875084cf-47e7-4173-b905-0b3607ac5dd4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:03.429497 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875084cf-47e7-4173-b905-0b3607ac5dd4 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:03.429987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875084cf-47e7-4173-b905-0b3607ac5dd4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:03.430494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875084cf-47e7-4173-b905-0b3607ac5dd4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:03.445587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 830/1661] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:03 2022] GET /identity/v3/auth/tokens => generated 3222 bytes in 278 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:03.762344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875084cf-47e7-4173-b905-0b3607ac5dd4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:03.762344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875084cf-47e7-4173-b905-0b3607ac5dd4 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:03.762344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 832/1662] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:03 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:03.962394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dac6d3-8221-4f50-8da6-8030505f0a7b None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:03.962763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dac6d3-8221-4f50-8da6-8030505f0a7b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:03.963120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90dac6d3-8221-4f50-8da6-8030505f0a7b None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:03.965035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 831/1663] 10.222.0.25 () {54 vars in 912 bytes} [Fri Jun 10 13:05:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:03.974598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac1db07-db5b-4b86-a5a5-164f81f8dba0 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:03.975136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac1db07-db5b-4b86-a5a5-164f81f8dba0 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:03.975246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac1db07-db5b-4b86-a5a5-164f81f8dba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:04.060867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-cac1db07-db5b-4b86-a5a5-164f81f8dba0 None None] MFA Rules not processed for user `d6794257c90e40bca2c8efd31b0c791a`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:04.061960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-cac1db07-db5b-4b86-a5a5-164f81f8dba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:04.129584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 833/1664] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:05:03 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:04.135452 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dac6d3-8221-4f50-8da6-8030505f0a7b None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:04.146561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:04.173369 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90dac6d3-8221-4f50-8da6-8030505f0a7b None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:04.192956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffad45d1-f46f-48bc-9a61-8b1ad5496e4a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:04.197838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffad45d1-f46f-48bc-9a61-8b1ad5496e4a service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:04.198192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffad45d1-f46f-48bc-9a61-8b1ad5496e4a service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:04.198537 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffad45d1-f46f-48bc-9a61-8b1ad5496e4a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:04.215945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffad45d1-f46f-48bc-9a61-8b1ad5496e4a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:04.218889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffad45d1-f46f-48bc-9a61-8b1ad5496e4a service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:04.240942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 832/1665] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:04 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:05:04.454531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac1db07-db5b-4b86-a5a5-164f81f8dba0 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:04.576110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac1db07-db5b-4b86-a5a5-164f81f8dba0 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:04.593909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:04.595389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:04.597282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:04.603847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] PATH_INFO: `/v3/users/56b679e9ccf94158b47bb3f32c30e322` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:04.603847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=56b679e9ccf94158b47bb3f32c30e322)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:04.603847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:04.985043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '56b679e9ccf94158b47bb3f32c30e322'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:04.986421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '56b679e9ccf94158b47bb3f32c30e322'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:04.997897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '56b679e9ccf94158b47bb3f32c30e322'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:05.169878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '56b679e9ccf94158b47bb3f32c30e322'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:05.170813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '56b679e9ccf94158b47bb3f32c30e322'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:05.214073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 834/1666] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:04 2022] DELETE /identity/v3/users/56b679e9ccf94158b47bb3f32c30e322 => generated 0 bytes in 757 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:05.218665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7cfb8a-174a-4a69-8be5-b7a015c953ef None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:05.219536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7cfb8a-174a-4a69-8be5-b7a015c953ef None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:05.219892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7cfb8a-174a-4a69-8be5-b7a015c953ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:05.274875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bd7cfb8a-174a-4a69-8be5-b7a015c953ef None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:05.276943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bd7cfb8a-174a-4a69-8be5-b7a015c953ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:05.420332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 833/1667] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:05.436392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:05.438505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:05.498262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7cfb8a-174a-4a69-8be5-b7a015c953ef None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:05.504864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bd7cfb8a-174a-4a69-8be5-b7a015c953ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:05.615472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb8ebd3-84c4-41b1-83be-856ea4b6ddd6 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:05.621023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:05.622475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:05.623201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:05.623654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:05.623911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7cfb8a-174a-4a69-8be5-b7a015c953ef None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:05.625926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:05.627903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3882a9-4e3e-4d94-a6cd-515aec5347cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:05.628811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3882a9-4e3e-4d94-a6cd-515aec5347cf service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:05.628938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3882a9-4e3e-4d94-a6cd-515aec5347cf service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:05.629168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3882a9-4e3e-4d94-a6cd-515aec5347cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:05.639488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3882a9-4e3e-4d94-a6cd-515aec5347cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:05.640986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3882a9-4e3e-4d94-a6cd-515aec5347cf service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:05.642149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:05.644023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:05.651609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 834/1668] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:05:05.659566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 835/1669] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:05 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:06.071972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:06.563244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:06.859933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b10378-6a50-4ab8-8c38-768644dbb571 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:06.860480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b10378-6a50-4ab8-8c38-768644dbb571 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:06.860992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b10378-6a50-4ab8-8c38-768644dbb571 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:06.916330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-46b10378-6a50-4ab8-8c38-768644dbb571 None None] MFA Rules not processed for user `d2776edb1dc342898968a3f9411caf9f`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:06.920446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-46b10378-6a50-4ab8-8c38-768644dbb571 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:06.938838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:06.941365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:06.981745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde94440-e9c1-420d-87ef-36005580ef7d service placement] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:07.008637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:07.009866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:07.010304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:07.011971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] PATH_INFO: `/v3/projects/2952f85640f14fbba1e3d7a3f784c521` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:07.014365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] RBAC: Authorizing `identity:delete_project(project_id=2952f85640f14fbba1e3d7a3f784c521)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:07.015561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: WARNING py.warnings [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:05:07.015561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: warnings.warn(msg) Jun 10 13:05:07.015561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: Jun 10 13:05:07.018820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:07.098071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 835/1670] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:06 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 240 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:07.242773 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:07.267110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2952f85640f14fbba1e3d7a3f784c521', 'request_id': 'req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:07.318000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2952f85640f14fbba1e3d7a3f784c521', 'request_id': 'req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:07.343400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2952f85640f14fbba1e3d7a3f784c521', 'request_id': 'req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:07.507340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] The token cache is being invalidate because project 2952f85640f14fbba1e3d7a3f784c521 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:07.516015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:07.516015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:07.516015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 836/1671] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:06 2022] DELETE /identity/v3/projects/2952f85640f14fbba1e3d7a3f784c521 => generated 0 bytes in 572 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:07.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b10378-6a50-4ab8-8c38-768644dbb571 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:07.535289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b10378-6a50-4ab8-8c38-768644dbb571 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:07.643322 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:07.644291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:07.644676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:07.645033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] PATH_INFO: `/v3/users/147e3838103847fd9d6852c8d43b09b6` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:07.646716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] RBAC: Authorizing `identity:delete_user(user_id=147e3838103847fd9d6852c8d43b09b6)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:07.647782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: WARNING py.warnings [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jun 10 13:05:07.647782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: warnings.warn(msg) Jun 10 13:05:07.647782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: Jun 10 13:05:07.649085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:07.793159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:07.796518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:07.828904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4bc2ce-6ffa-4c04-8f15-619955a0f14c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:07.837116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:07.838783 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:07.839928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:07.840386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] PATH_INFO: `/v3/users/dad62ca49b6f49c3b6f084accc6bc2f1` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:07.842382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=dad62ca49b6f49c3b6f084accc6bc2f1)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:07.845644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:07.926435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '147e3838103847fd9d6852c8d43b09b6'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:07.927222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '147e3838103847fd9d6852c8d43b09b6'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:07.927948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '147e3838103847fd9d6852c8d43b09b6'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:07.944087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '147e3838103847fd9d6852c8d43b09b6'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:08.029051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8013a897-2cfc-4f2f-98ff-63fe64dea57d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '147e3838103847fd9d6852c8d43b09b6'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:08.124881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 836/1672] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:07 2022] DELETE /identity/v3/users/147e3838103847fd9d6852c8d43b09b6 => generated 0 bytes in 606 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:08.131968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dad62ca49b6f49c3b6f084accc6bc2f1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:08.133382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dad62ca49b6f49c3b6f084accc6bc2f1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:08.142940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dad62ca49b6f49c3b6f084accc6bc2f1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:08.233066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dad62ca49b6f49c3b6f084accc6bc2f1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:08.234795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dad62ca49b6f49c3b6f084accc6bc2f1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:08.280367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 837/1673] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:07 2022] DELETE /identity/v3/users/dad62ca49b6f49c3b6f084accc6bc2f1 => generated 0 bytes in 485 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:08.282659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9260f195-cc6b-4270-b05b-fb1847ae9ce3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:08.282970 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9260f195-cc6b-4270-b05b-fb1847ae9ce3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:08.283260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9260f195-cc6b-4270-b05b-fb1847ae9ce3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:08.290765 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:08.338521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9260f195-cc6b-4270-b05b-fb1847ae9ce3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:08.338521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9260f195-cc6b-4270-b05b-fb1847ae9ce3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:08.501140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 837/1674] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 219 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:08.638009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:08.639822 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:08.750161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72111ddf-5942-4b68-aba7-53f44257c8cf admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:08.758681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3983db-55db-4940-9e2f-8e7e19c2c054 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:08.759813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3983db-55db-4940-9e2f-8e7e19c2c054 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:08.763610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3983db-55db-4940-9e2f-8e7e19c2c054 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:08.764206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3983db-55db-4940-9e2f-8e7e19c2c054 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:08.786759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3983db-55db-4940-9e2f-8e7e19c2c054 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:08.789553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3983db-55db-4940-9e2f-8e7e19c2c054 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:08.806401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 838/1675] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:09.009411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9260f195-cc6b-4270-b05b-fb1847ae9ce3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:09.010845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9260f195-cc6b-4270-b05b-fb1847ae9ce3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:09.013815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7669436a-22d4-4884-a68a-609f5f5ae546 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:09.014114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7669436a-22d4-4884-a68a-609f5f5ae546 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:09.014398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7669436a-22d4-4884-a68a-609f5f5ae546 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:09.017378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 839/1676] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 13:05:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:09.031930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a285cfb-7361-487d-9452-da6dc5b5cbce None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:09.032586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a285cfb-7361-487d-9452-da6dc5b5cbce None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:09.033034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a285cfb-7361-487d-9452-da6dc5b5cbce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:09.034656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9260f195-cc6b-4270-b05b-fb1847ae9ce3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:09.065664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:09.067575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:09.068019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:09.068446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] PATH_INFO: `/v3/users/c1cd93116d3e46d998f49cace4fdab17` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:09.070393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1cd93116d3e46d998f49cace4fdab17)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:09.077163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:09.088810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5a285cfb-7361-487d-9452-da6dc5b5cbce None None] MFA Rules not processed for user `0c315cfb1aa44f3c9a755f3cb8e5cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:09.090056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5a285cfb-7361-487d-9452-da6dc5b5cbce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:09.231901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 840/1677] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 13:05:09 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Jun 10 13:05:09.233456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:09.255064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8257c484-ad62-4bfe-ac31-ce78a1e3568c None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:09.255597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8257c484-ad62-4bfe-ac31-ce78a1e3568c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:09.255878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8257c484-ad62-4bfe-ac31-ce78a1e3568c None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:09.257583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 841/1678] 10.222.0.25 () {54 vars in 960 bytes} [Fri Jun 10 13:05:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:09.263715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff8b88-30e0-4d48-b39e-e4c744d5c28f None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:09.264193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff8b88-30e0-4d48-b39e-e4c744d5c28f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:09.264611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff8b88-30e0-4d48-b39e-e4c744d5c28f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:09.302740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6cff8b88-30e0-4d48-b39e-e4c744d5c28f None None] MFA Rules not processed for user `0c315cfb1aa44f3c9a755f3cb8e5cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:09.304470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6cff8b88-30e0-4d48-b39e-e4c744d5c28f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:09.314019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 842/1679] 10.222.0.25 () {58 vars in 1057 bytes} [Fri Jun 10 13:05:09 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:09.329969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff8b88-30e0-4d48-b39e-e4c744d5c28f None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:09.347080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff8b88-30e0-4d48-b39e-e4c744d5c28f None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:09.363949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6689137-77b8-4dba-a674-d945bfa028e1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:09.365095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6689137-77b8-4dba-a674-d945bfa028e1 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:09.365388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6689137-77b8-4dba-a674-d945bfa028e1 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:09.365666 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6689137-77b8-4dba-a674-d945bfa028e1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:09.402754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6689137-77b8-4dba-a674-d945bfa028e1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:09.404415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6689137-77b8-4dba-a674-d945bfa028e1 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:09.407260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1cd93116d3e46d998f49cace4fdab17'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:09.408573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1cd93116d3e46d998f49cace4fdab17'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:09.410015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1cd93116d3e46d998f49cace4fdab17'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:09.429234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 843/1680] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:09 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:05:09.439753 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1cd93116d3e46d998f49cace4fdab17'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:09.611163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1cd93116d3e46d998f49cace4fdab17'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:09.664269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 838/1681] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:09 2022] DELETE /identity/v3/users/c1cd93116d3e46d998f49cace4fdab17 => generated 0 bytes in 655 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:09.669398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d793dc-65c1-489d-bba6-e8e6f03636be None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:09.669656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d793dc-65c1-489d-bba6-e8e6f03636be None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:09.669938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24d793dc-65c1-489d-bba6-e8e6f03636be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:09.728836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-24d793dc-65c1-489d-bba6-e8e6f03636be None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:09.732478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-24d793dc-65c1-489d-bba6-e8e6f03636be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:09.895374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 844/1682] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:09.951720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:09.952777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:09.954189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:10.095086 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b80e4466-c85b-49c6-87da-296a2347b6cd admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:10.099299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0d75be-5671-42d5-b1c4-e1a97b6c4dfb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:10.099968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0d75be-5671-42d5-b1c4-e1a97b6c4dfb service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:10.100634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0d75be-5671-42d5-b1c4-e1a97b6c4dfb service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:10.100634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0d75be-5671-42d5-b1c4-e1a97b6c4dfb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:10.137777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0d75be-5671-42d5-b1c4-e1a97b6c4dfb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:10.145919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0d75be-5671-42d5-b1c4-e1a97b6c4dfb service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:10.173879 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 839/1683] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:05:10.754216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8eed20-5d13-4db2-ba49-1c6541bfc125 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:10.754735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8eed20-5d13-4db2-ba49-1c6541bfc125 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:10.754831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8eed20-5d13-4db2-ba49-1c6541bfc125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:10.782532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ad8eed20-5d13-4db2-ba49-1c6541bfc125 None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:10.786003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ad8eed20-5d13-4db2-ba49-1c6541bfc125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:10.815703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0d75be-5671-42d5-b1c4-e1a97b6c4dfb service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:10.817428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-db0d75be-5671-42d5-b1c4-e1a97b6c4dfb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:10.840149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0d75be-5671-42d5-b1c4-e1a97b6c4dfb service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:10.852542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:10.853474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:10.853761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:10.854040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] PATH_INFO: `/v3/projects/724ce3cd23ae46eb934b04eb947f4236` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:10.855329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] RBAC: Authorizing `identity:delete_project(project_id=724ce3cd23ae46eb934b04eb947f4236)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:10.860241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:10.940229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 845/1684] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:05:10 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:10.942367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8eed20-5d13-4db2-ba49-1c6541bfc125 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:10.946134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ad8eed20-5d13-4db2-ba49-1c6541bfc125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:10.968952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8eed20-5d13-4db2-ba49-1c6541bfc125 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:10.996234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:10.997390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:10.997876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:10.998338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] PATH_INFO: `/v3/projects/c40c1b8609d249bfb31e3c9bd5cc0623` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.000058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c40c1b8609d249bfb31e3c9bd5cc0623)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:11.002270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:11.154121 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '724ce3cd23ae46eb934b04eb947f4236', 'request_id': 'req-bb8ea384-783d-47b9-97b0-75cd4e22cb41'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.235924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c40c1b8609d249bfb31e3c9bd5cc0623', 'request_id': 'req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.253183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c40c1b8609d249bfb31e3c9bd5cc0623', 'request_id': 'req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.265509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c40c1b8609d249bfb31e3c9bd5cc0623', 'request_id': 'req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.329090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '724ce3cd23ae46eb934b04eb947f4236', 'request_id': 'req-bb8ea384-783d-47b9-97b0-75cd4e22cb41'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.344664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '724ce3cd23ae46eb934b04eb947f4236', 'request_id': 'req-bb8ea384-783d-47b9-97b0-75cd4e22cb41'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.372227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] The token cache is being invalidate because project 724ce3cd23ae46eb934b04eb947f4236 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:11.372735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.373585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bb8ea384-783d-47b9-97b0-75cd4e22cb41 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.379692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 840/1685] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:10 2022] DELETE /identity/v3/projects/724ce3cd23ae46eb934b04eb947f4236 => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:11.385145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcbca4a-93ac-4448-9ac3-191040ca405d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:11.385145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcbca4a-93ac-4448-9ac3-191040ca405d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:11.385145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bcbca4a-93ac-4448-9ac3-191040ca405d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.403067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9bcbca4a-93ac-4448-9ac3-191040ca405d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:11.404303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9bcbca4a-93ac-4448-9ac3-191040ca405d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:11.454680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] The token cache is being invalidate because project c40c1b8609d249bfb31e3c9bd5cc0623 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:11.454984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.455657 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2f8afdb1-ed41-4a3f-97c1-e29b63ec6160 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:11.457527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 846/1686] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:10 2022] DELETE /identity/v3/projects/c40c1b8609d249bfb31e3c9bd5cc0623 => generated 0 bytes in 517 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:11.462284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1de489-9296-4842-80e1-81c81ea90032 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:11.462792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1de489-9296-4842-80e1-81c81ea90032 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:11.463236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1de489-9296-4842-80e1-81c81ea90032 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.490579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 841/1687] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:11.491607 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcbca4a-93ac-4448-9ac3-191040ca405d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:11.494612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9bcbca4a-93ac-4448-9ac3-191040ca405d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:11.500908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6a1de489-9296-4842-80e1-81c81ea90032 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:11.502155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6a1de489-9296-4842-80e1-81c81ea90032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:11.520797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 847/1688] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:11.523244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6a51e6-ade6-4f50-afd5-ef86e20dae21 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:11.523530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6a51e6-ade6-4f50-afd5-ef86e20dae21 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:11.524128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6a51e6-ade6-4f50-afd5-ef86e20dae21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.528582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:11.569112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3f6a51e6-ade6-4f50-afd5-ef86e20dae21 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:11.571024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3f6a51e6-ade6-4f50-afd5-ef86e20dae21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:11.588508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 848/1689] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:11.588508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7c28e0-8efc-43bf-92e8-eb15ae0d69a5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:11.588508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7c28e0-8efc-43bf-92e8-eb15ae0d69a5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:11.588857 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7c28e0-8efc-43bf-92e8-eb15ae0d69a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.627720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bcbca4a-93ac-4448-9ac3-191040ca405d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:11.652119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-fa7c28e0-8efc-43bf-92e8-eb15ae0d69a5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:11.656921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92803325-4e02-4de0-83f3-e1f56eab9521 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:11.657601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92803325-4e02-4de0-83f3-e1f56eab9521 service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:11.658805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92803325-4e02-4de0-83f3-e1f56eab9521 service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:11.659375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92803325-4e02-4de0-83f3-e1f56eab9521 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.661956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-fa7c28e0-8efc-43bf-92e8-eb15ae0d69a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:11.666062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-92803325-4e02-4de0-83f3-e1f56eab9521 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:11.695181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 849/1690] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:11.702166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7c28e0-8efc-43bf-92e8-eb15ae0d69a5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:11.713539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7c28e0-8efc-43bf-92e8-eb15ae0d69a5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:11.719145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c49b32c-f034-4eac-b0de-73c0ac2b0f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:11.727507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c49b32c-f034-4eac-b0de-73c0ac2b0f47 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:11.727970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c49b32c-f034-4eac-b0de-73c0ac2b0f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:11.728481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c49b32c-f034-4eac-b0de-73c0ac2b0f47 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.730112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c49b32c-f034-4eac-b0de-73c0ac2b0f47 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:11.732589 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c49b32c-f034-4eac-b0de-73c0ac2b0f47 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:11.733235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c49b32c-f034-4eac-b0de-73c0ac2b0f47 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:11.758130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 850/1691] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:11.761388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c49b32c-f034-4eac-b0de-73c0ac2b0f47 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:11.783752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c49b32c-f034-4eac-b0de-73c0ac2b0f47 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:11.799547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:11.800666 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:11.801706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:11.802606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.803881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:11.805533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:11.807558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92803325-4e02-4de0-83f3-e1f56eab9521 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:11.811988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92803325-4e02-4de0-83f3-e1f56eab9521 service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:11.812178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:11.843587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 842/1692] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:05:11 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 13:05:11.848072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92803325-4e02-4de0-83f3-e1f56eab9521 service swift] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:11.851745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 851/1693] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:11.852863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:11.856529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:11.878083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92803325-4e02-4de0-83f3-e1f56eab9521 service swift] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:11.879110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec45cdb-e107-47ba-8828-1cde3a4f62a9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:11.887934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:11.887934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:11.887934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:11.887934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] PATH_INFO: `/v3/projects/77208dd471cf4142b529ce6c6cb7f618` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.889505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] RBAC: Authorizing `identity:delete_project(project_id=77208dd471cf4142b529ce6c6cb7f618)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:11.892617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:11.906598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba471e25-2a23-45e8-82f2-6af609894dd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:11.908115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba471e25-2a23-45e8-82f2-6af609894dd2 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:11.908494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba471e25-2a23-45e8-82f2-6af609894dd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:11.908865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba471e25-2a23-45e8-82f2-6af609894dd2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:11.909783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba471e25-2a23-45e8-82f2-6af609894dd2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:11.913992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba471e25-2a23-45e8-82f2-6af609894dd2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:12.084915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 843/1694] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:11 2022] POST /identity/v3/projects => generated 381 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:12.088826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba471e25-2a23-45e8-82f2-6af609894dd2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:12.135517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77208dd471cf4142b529ce6c6cb7f618', 'request_id': 'req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:12.155962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77208dd471cf4142b529ce6c6cb7f618', 'request_id': 'req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:12.165246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77208dd471cf4142b529ce6c6cb7f618', 'request_id': 'req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:12.220140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:12.288819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba471e25-2a23-45e8-82f2-6af609894dd2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:12.341867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] The token cache is being invalidate because project 77208dd471cf4142b529ce6c6cb7f618 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:12.342351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:12.343761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:12.345588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 852/1695] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:11 2022] DELETE /identity/v3/projects/77208dd471cf4142b529ce6c6cb7f618 => generated 0 bytes in 493 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:12.349981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:12.354687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:12.534008 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c529c-78ad-48a0-af66-c31d789ba47c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:12.535337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c529c-78ad-48a0-af66-c31d789ba47c admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:12.535634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c529c-78ad-48a0-af66-c31d789ba47c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:12.538324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c529c-78ad-48a0-af66-c31d789ba47c admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:12.539465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c529c-78ad-48a0-af66-c31d789ba47c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:12.547811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c529c-78ad-48a0-af66-c31d789ba47c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:12.598202 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3554517c-8e35-4f6c-8ef1-4c3fc2fd2f5a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:12.615915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1abbcd-feec-4a5a-b5a9-4807b748b6e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:12.616789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1abbcd-feec-4a5a-b5a9-4807b748b6e2 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:12.617113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1abbcd-feec-4a5a-b5a9-4807b748b6e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:12.618801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1abbcd-feec-4a5a-b5a9-4807b748b6e2 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:12.619973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1abbcd-feec-4a5a-b5a9-4807b748b6e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:12.623106 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1abbcd-feec-4a5a-b5a9-4807b748b6e2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:12.686486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 844/1696] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:12 2022] POST /identity/v3/projects => generated 367 bytes in 598 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:12.690318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c529c-78ad-48a0-af66-c31d789ba47c admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:12.692227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8a8c529c-78ad-48a0-af66-c31d789ba47c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:12.756721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 853/1697] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:12 2022] POST /identity/v3/users => generated 291 bytes in 401 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:12.763012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7712bb9-df08-4280-8ebb-adddb4b2947c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:12.763732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7712bb9-df08-4280-8ebb-adddb4b2947c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:12.763732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7712bb9-df08-4280-8ebb-adddb4b2947c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:12.913273 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-f7712bb9-df08-4280-8ebb-adddb4b2947c None None] MFA Rules not processed for user `b1d1712ea523498f92f258383863b906`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:12.916656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f7712bb9-df08-4280-8ebb-adddb4b2947c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:12.932881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c529c-78ad-48a0-af66-c31d789ba47c admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:12.952951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:12.957748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:12.960192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:12.960371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] PATH_INFO: `/v3/users/e748601d3d2f48449abfd1bd384c5126` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:12.968538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] RBAC: Authorizing `identity:delete_user(user_id=e748601d3d2f48449abfd1bd384c5126)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:12.974700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:13.071014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 854/1698] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:12 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:13.071710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7712bb9-df08-4280-8ebb-adddb4b2947c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:13.073199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f7712bb9-df08-4280-8ebb-adddb4b2947c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:13.089335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7712bb9-df08-4280-8ebb-adddb4b2947c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:13.095199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec62dc-851d-4107-aa5d-ed29b604f89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:13.096078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec62dc-851d-4107-aa5d-ed29b604f89d admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:13.096534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec62dc-851d-4107-aa5d-ed29b604f89d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:13.097153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ec62dc-851d-4107-aa5d-ed29b604f89d admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:13.097989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec62dc-851d-4107-aa5d-ed29b604f89d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:13.100518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ec62dc-851d-4107-aa5d-ed29b604f89d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:13.312176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 855/1699] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:13 2022] POST /identity/v3/users => generated 284 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:13.313242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec62dc-851d-4107-aa5d-ed29b604f89d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:13.345085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ec62dc-851d-4107-aa5d-ed29b604f89d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:13.346273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e748601d3d2f48449abfd1bd384c5126'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:13.348544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e748601d3d2f48449abfd1bd384c5126'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:13.349516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e748601d3d2f48449abfd1bd384c5126'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:13.359653 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c896f1c-d81c-48c1-8e65-9a3cad8daf77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:13.360476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c896f1c-d81c-48c1-8e65-9a3cad8daf77 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:13.360884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c896f1c-d81c-48c1-8e65-9a3cad8daf77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:13.361067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c896f1c-d81c-48c1-8e65-9a3cad8daf77 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:13.362055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c896f1c-d81c-48c1-8e65-9a3cad8daf77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:13.363841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e748601d3d2f48449abfd1bd384c5126'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:13.363989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c896f1c-d81c-48c1-8e65-9a3cad8daf77 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:13.389263 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 856/1700] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:13 2022] GET /identity/v3/roles => generated 1420 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:13.392031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c896f1c-d81c-48c1-8e65-9a3cad8daf77 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:13.398570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6c896f1c-d81c-48c1-8e65-9a3cad8daf77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:13.488141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e748601d3d2f48449abfd1bd384c5126'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:13.522136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 845/1701] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:12 2022] DELETE /identity/v3/users/e748601d3d2f48449abfd1bd384c5126 => generated 0 bytes in 835 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:13.524503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:13.533805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:13.608573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=87,keystone:INSERT=5,keystone:DELETE=8 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:13.618102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c896f1c-d81c-48c1-8e65-9a3cad8daf77 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:13.703021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:13.717735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b53973-4c62-4c28-af95-c1dde33a93de admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:13.741329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30926e4d-db50-4120-a1f2-d4ab665dc85a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:13.742206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30926e4d-db50-4120-a1f2-d4ab665dc85a service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:13.742521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30926e4d-db50-4120-a1f2-d4ab665dc85a service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:13.742809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30926e4d-db50-4120-a1f2-d4ab665dc85a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:13.745290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-30926e4d-db50-4120-a1f2-d4ab665dc85a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:13.763536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaed09-c644-4284-b9ea-4bffba81a6bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:13.770673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaed09-c644-4284-b9ea-4bffba81a6bb admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:13.773443 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaed09-c644-4284-b9ea-4bffba81a6bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:13.775410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18aaed09-c644-4284-b9ea-4bffba81a6bb admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:13.781057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18aaed09-c644-4284-b9ea-4bffba81a6bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:13.786945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18aaed09-c644-4284-b9ea-4bffba81a6bb admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:13.843484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 846/1702] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:13 2022] GET /identity/v3/roles => generated 1420 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:13.845363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaed09-c644-4284-b9ea-4bffba81a6bb admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:13.850087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30926e4d-db50-4120-a1f2-d4ab665dc85a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:13.854560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-18aaed09-c644-4284-b9ea-4bffba81a6bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:13.854687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30926e4d-db50-4120-a1f2-d4ab665dc85a service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:13.862612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 857/1703] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:13 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 473 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:13.864337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5be556-ab80-45e0-83a8-7aba6c2d9372 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:13.865172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5be556-ab80-45e0-83a8-7aba6c2d9372 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:13.865878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c5be556-ab80-45e0-83a8-7aba6c2d9372 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:13.867636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 858/1704] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 13:05:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:13.880835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5be556-ab80-45e0-83a8-7aba6c2d9372 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:13.910257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c5be556-ab80-45e0-83a8-7aba6c2d9372 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:13.920666 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dfa62d-de82-4934-94bf-72c03ada52e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:13.925960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dfa62d-de82-4934-94bf-72c03ada52e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:13.925960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dfa62d-de82-4934-94bf-72c03ada52e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:13.925960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3dfa62d-de82-4934-94bf-72c03ada52e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:13.925960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dfa62d-de82-4934-94bf-72c03ada52e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:13.926683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18aaed09-c644-4284-b9ea-4bffba81a6bb admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:13.926776 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3dfa62d-de82-4934-94bf-72c03ada52e4 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:13.941521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 859/1705] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:13 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:13.942277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c44aea-f13e-4322-9188-33d79bdb3b0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:13.942417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dfa62d-de82-4934-94bf-72c03ada52e4 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:13.942992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c44aea-f13e-4322-9188-33d79bdb3b0d admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:13.943366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c44aea-f13e-4322-9188-33d79bdb3b0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:13.943558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8c44aea-f13e-4322-9188-33d79bdb3b0d admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:13.944404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c44aea-f13e-4322-9188-33d79bdb3b0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:13.944648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a3dfa62d-de82-4934-94bf-72c03ada52e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:13.957245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8c44aea-f13e-4322-9188-33d79bdb3b0d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:13.959493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3dfa62d-de82-4934-94bf-72c03ada52e4 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:13.965751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:13.967430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:13.967430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:13.967430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] PATH_INFO: `/v3/users/fbe30e8af9324b0e869fc6cf5f155728` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:13.968889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fbe30e8af9324b0e869fc6cf5f155728)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:13.974247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:13.991850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 847/1706] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:13 2022] GET /identity/v3/roles => generated 1420 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:13.993020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c44aea-f13e-4322-9188-33d79bdb3b0d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:14.017944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8c44aea-f13e-4322-9188-33d79bdb3b0d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:14.024411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08113e90-8b61-426f-a791-e274100e24f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:14.028962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08113e90-8b61-426f-a791-e274100e24f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:14.028962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08113e90-8b61-426f-a791-e274100e24f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:14.028962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08113e90-8b61-426f-a791-e274100e24f9 admin admin] PATH_INFO: `/v3/projects/ecf352f2fcaf489b8fed13a63da668d9/users/dcdbde8f62d142949bd966d6e3089bff/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:14.045500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08113e90-8b61-426f-a791-e274100e24f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ecf352f2fcaf489b8fed13a63da668d9, user_id=dcdbde8f62d142949bd966d6e3089bff, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:14.049793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08113e90-8b61-426f-a791-e274100e24f9 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:14.353531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 848/1707] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:13 2022] PUT /identity/v3/projects/ecf352f2fcaf489b8fed13a63da668d9/users/dcdbde8f62d142949bd966d6e3089bff/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:14.356827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a663727e-bd65-4583-a246-5a3289f816e3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:14.357259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a663727e-bd65-4583-a246-5a3289f816e3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:14.357584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a663727e-bd65-4583-a246-5a3289f816e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:14.448119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a663727e-bd65-4583-a246-5a3289f816e3 None None] MFA Rules not processed for user `0c315cfb1aa44f3c9a755f3cb8e5cd5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:14.448119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a663727e-bd65-4583-a246-5a3289f816e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:14.542956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fbe30e8af9324b0e869fc6cf5f155728'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:14.545539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fbe30e8af9324b0e869fc6cf5f155728'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:14.556010 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 849/1708] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 13:05:14 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 201 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:14.564689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fbe30e8af9324b0e869fc6cf5f155728'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:14.580083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a663727e-bd65-4583-a246-5a3289f816e3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:14.592477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a663727e-bd65-4583-a246-5a3289f816e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:14.734112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fbe30e8af9324b0e869fc6cf5f155728'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:14.734824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-aa257eab-aa3c-4eb5-a6bd-f189267070d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fbe30e8af9324b0e869fc6cf5f155728'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:14.785037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 860/1709] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:13 2022] DELETE /identity/v3/users/fbe30e8af9324b0e869fc6cf5f155728 => generated 0 bytes in 843 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:14.788486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b45b97-9fff-4009-8ea9-1b505c7669e5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:14.789910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b45b97-9fff-4009-8ea9-1b505c7669e5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:14.790449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b45b97-9fff-4009-8ea9-1b505c7669e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:14.832289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a663727e-bd65-4583-a246-5a3289f816e3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:14.870994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-05b45b97-9fff-4009-8ea9-1b505c7669e5 None None] MFA Rules not processed for user `dcdbde8f62d142949bd966d6e3089bff`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:14.874230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-05b45b97-9fff-4009-8ea9-1b505c7669e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:15.073553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d44f7bc-12fb-41c3-b7de-1db22720495a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:15.075644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d44f7bc-12fb-41c3-b7de-1db22720495a admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:15.076902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d44f7bc-12fb-41c3-b7de-1db22720495a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:15.077429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d44f7bc-12fb-41c3-b7de-1db22720495a admin admin] PATH_INFO: `/v3/projects/8f784f9f33f34372888b4575b5b42771/users/fc9e0f8a8876497687fd7eab5795f31e/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:15.110647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 861/1710] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:14 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:15.111619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b45b97-9fff-4009-8ea9-1b505c7669e5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:15.121481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-05b45b97-9fff-4009-8ea9-1b505c7669e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:15.126637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d44f7bc-12fb-41c3-b7de-1db22720495a admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f784f9f33f34372888b4575b5b42771, user_id=fc9e0f8a8876497687fd7eab5795f31e, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:15.136543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d44f7bc-12fb-41c3-b7de-1db22720495a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:15.273298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b45b97-9fff-4009-8ea9-1b505c7669e5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:15.290374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71280831-8735-46e6-9896-60d0a22f62a2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:15.291780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71280831-8735-46e6-9896-60d0a22f62a2 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:15.293118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71280831-8735-46e6-9896-60d0a22f62a2 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:15.315101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71280831-8735-46e6-9896-60d0a22f62a2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:15.315101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71280831-8735-46e6-9896-60d0a22f62a2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:15.315101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71280831-8735-46e6-9896-60d0a22f62a2 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:15.317139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 862/1711] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:15 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:15.318581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315b8721-b7b0-4365-b908-2fd14a959fa9 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:15.318898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315b8721-b7b0-4365-b908-2fd14a959fa9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:15.319158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315b8721-b7b0-4365-b908-2fd14a959fa9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:15.347867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-315b8721-b7b0-4365-b908-2fd14a959fa9 None None] MFA Rules not processed for user `d6794257c90e40bca2c8efd31b0c791a`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:15.349469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-315b8721-b7b0-4365-b908-2fd14a959fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:15.390138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 850/1712] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:14 2022] PUT /identity/v3/projects/8f784f9f33f34372888b4575b5b42771/users/fc9e0f8a8876497687fd7eab5795f31e/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 833 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:15.390744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d44f7bc-12fb-41c3-b7de-1db22720495a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:15.396923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2d44f7bc-12fb-41c3-b7de-1db22720495a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:15.515070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 863/1713] 10.222.0.24 () {58 vars in 1015 bytes} [Fri Jun 10 13:05:15 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:15.516017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315b8721-b7b0-4365-b908-2fd14a959fa9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:15.525405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-315b8721-b7b0-4365-b908-2fd14a959fa9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:15.547256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d44f7bc-12fb-41c3-b7de-1db22720495a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:15.556246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7348c1-530a-421e-bea3-dacf5cabe293 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:15.557651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7348c1-530a-421e-bea3-dacf5cabe293 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:15.558112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7348c1-530a-421e-bea3-dacf5cabe293 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:15.559229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7348c1-530a-421e-bea3-dacf5cabe293 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:15.579618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315b8721-b7b0-4365-b908-2fd14a959fa9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:15.747101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7348c1-530a-421e-bea3-dacf5cabe293 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:15.750765 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca7348c1-530a-421e-bea3-dacf5cabe293 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:15.762930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:15.763952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:15.764239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:15.764527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] PATH_INFO: `/v3/projects/cccd79a54d644360890f79e164e18fc3` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:15.772705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cccd79a54d644360890f79e164e18fc3)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:15.777619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:15.812299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 851/1714] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:15 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:15.818484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38da37b-f9be-4764-b492-92d116fd70dd None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:15.820121 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38da37b-f9be-4764-b492-92d116fd70dd None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:15.822951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38da37b-f9be-4764-b492-92d116fd70dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:15.863462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e38da37b-f9be-4764-b492-92d116fd70dd None None] MFA Rules not processed for user `9c8b2c27b69a4776ab6b7b7820da02d9`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:15.865175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e38da37b-f9be-4764-b492-92d116fd70dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:15.988221 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cccd79a54d644360890f79e164e18fc3', 'request_id': 'req-d3294138-883b-452d-8e2c-0690b1d46fe8'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.019704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:16.029734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cccd79a54d644360890f79e164e18fc3', 'request_id': 'req-d3294138-883b-452d-8e2c-0690b1d46fe8'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.058052 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cccd79a54d644360890f79e164e18fc3', 'request_id': 'req-d3294138-883b-452d-8e2c-0690b1d46fe8'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.065448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:16.139289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 852/1715] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:15 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:16.140879 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081b5ed1-fad2-4ee2-adc6-e010622a6875 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:16.141183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081b5ed1-fad2-4ee2-adc6-e010622a6875 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:16.141472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081b5ed1-fad2-4ee2-adc6-e010622a6875 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:16.142227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] The token cache is being invalidate because project cccd79a54d644360890f79e164e18fc3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:16.142534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.145173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.153201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 864/1716] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:15 2022] DELETE /identity/v3/projects/cccd79a54d644360890f79e164e18fc3 => generated 0 bytes in 634 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:16.154275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:16.156289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:16.227953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-081b5ed1-fad2-4ee2-adc6-e010622a6875 None None] MFA Rules not processed for user `fc9e0f8a8876497687fd7eab5795f31e`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:16.231619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-081b5ed1-fad2-4ee2-adc6-e010622a6875 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:16.257543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3294138-883b-452d-8e2c-0690b1d46fe8 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:16.272938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:16.273961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:16.274445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:16.274959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:16.278005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:16.300171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:16.302144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:16.332874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 865/1717] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:16 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:16.334334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:16.339428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:16.483620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 853/1718] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:16 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 344 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:16.485402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081b5ed1-fad2-4ee2-adc6-e010622a6875 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:16.490878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-081b5ed1-fad2-4ee2-adc6-e010622a6875 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:16.541580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5843905-ffbf-4abd-b1de-220581964c5a service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:16.554366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:16.555602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:16.556045 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:16.556434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] PATH_INFO: `/v3/projects/340a9de2a2fb41018431aab744493b31` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:16.557833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] RBAC: Authorizing `identity:delete_project(project_id=340a9de2a2fb41018431aab744493b31)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:16.583721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:16.715893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081b5ed1-fad2-4ee2-adc6-e010622a6875 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:16.725675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:16.726665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:16.726976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:16.727262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:16.747423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:16.760181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '340a9de2a2fb41018431aab744493b31', 'request_id': 'req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.776329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '340a9de2a2fb41018431aab744493b31', 'request_id': 'req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.797353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '340a9de2a2fb41018431aab744493b31', 'request_id': 'req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.918484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] The token cache is being invalidate because project 340a9de2a2fb41018431aab744493b31 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:16.918789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.919912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1ef62835-d9e5-48f7-bf0f-a779d5c6242c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:16.922275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 866/1719] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:16 2022] DELETE /identity/v3/projects/340a9de2a2fb41018431aab744493b31 => generated 0 bytes in 588 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:16.928708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0b5cf-be29-4e73-8eaa-33733c7806d7 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:16.929420 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0b5cf-be29-4e73-8eaa-33733c7806d7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:16.930565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0b5cf-be29-4e73-8eaa-33733c7806d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:17.004202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:17.006354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:17.018726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3cd0b5cf-be29-4e73-8eaa-33733c7806d7 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:17.020442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3cd0b5cf-be29-4e73-8eaa-33733c7806d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:17.049740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 854/1720] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:16 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 565 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:17.051436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:17.065339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:17.110834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26e0f18-2e56-45de-82e0-c7dd3eb22ed5 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:17.304832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 867/1721] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 355 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:17.304832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd0b5cf-be29-4e73-8eaa-33733c7806d7 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:17.304832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3cd0b5cf-be29-4e73-8eaa-33733c7806d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:17.314362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd0b5cf-be29-4e73-8eaa-33733c7806d7 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:17.330066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:17.333067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:17.339519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:17.339519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] PATH_INFO: `/v3/users/d2776edb1dc342898968a3f9411caf9f` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:17.339519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=d2776edb1dc342898968a3f9411caf9f)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:17.346294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:17.356708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f28f8b-6eb4-4ead-b1e6-f38b3f394905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:17.357710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f28f8b-6eb4-4ead-b1e6-f38b3f394905 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:17.358224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f28f8b-6eb4-4ead-b1e6-f38b3f394905 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:17.358688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f28f8b-6eb4-4ead-b1e6-f38b3f394905 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:17.359608 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f28f8b-6eb4-4ead-b1e6-f38b3f394905 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:17.361872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f28f8b-6eb4-4ead-b1e6-f38b3f394905 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:17.369261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:17.449270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:DELETE=10,keystone:SELECT=88,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:17.500192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 855/1722] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:17 2022] POST /identity/v3/projects => generated 381 bytes in 449 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:17.510919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b101c-96dd-442f-a163-4e93baad89c0 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:17.511246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b101c-96dd-442f-a163-4e93baad89c0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:17.511539 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223b101c-96dd-442f-a163-4e93baad89c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:17.572220 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-223b101c-96dd-442f-a163-4e93baad89c0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:17.587344 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-223b101c-96dd-442f-a163-4e93baad89c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:17.673738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd2776edb1dc342898968a3f9411caf9f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:17.673738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd2776edb1dc342898968a3f9411caf9f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:17.750498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd2776edb1dc342898968a3f9411caf9f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:17.824679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 856/1723] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 320 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:17.825939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b101c-96dd-442f-a163-4e93baad89c0 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:17.829097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-223b101c-96dd-442f-a163-4e93baad89c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:17.863924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223b101c-96dd-442f-a163-4e93baad89c0 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:17.871327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98daaf19-5f6b-4416-b9fa-99aa9dc2a9df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:17.873439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98daaf19-5f6b-4416-b9fa-99aa9dc2a9df admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:17.880243 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98daaf19-5f6b-4416-b9fa-99aa9dc2a9df admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:17.881067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98daaf19-5f6b-4416-b9fa-99aa9dc2a9df admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:17.881900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98daaf19-5f6b-4416-b9fa-99aa9dc2a9df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:17.884513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98daaf19-5f6b-4416-b9fa-99aa9dc2a9df admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:17.920046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd2776edb1dc342898968a3f9411caf9f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:17.920793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd2776edb1dc342898968a3f9411caf9f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:17.948062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 868/1724] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:17 2022] DELETE /identity/v3/users/d2776edb1dc342898968a3f9411caf9f => generated 0 bytes in 654 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:17.948062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:18.017888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 857/1725] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:17 2022] POST /identity/v3/users => generated 297 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:18.018856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98daaf19-5f6b-4416-b9fa-99aa9dc2a9df admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:18.031190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98daaf19-5f6b-4416-b9fa-99aa9dc2a9df admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:18.085084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bd65e7-894b-4653-b898-1f876a8545ba admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:18.097591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cecce74-4f35-4309-94e5-9fe9fe4074a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:18.098280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cecce74-4f35-4309-94e5-9fe9fe4074a4 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:18.104941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cecce74-4f35-4309-94e5-9fe9fe4074a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:18.104941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cecce74-4f35-4309-94e5-9fe9fe4074a4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:18.104941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cecce74-4f35-4309-94e5-9fe9fe4074a4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:18.104941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cecce74-4f35-4309-94e5-9fe9fe4074a4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:18.104941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cecce74-4f35-4309-94e5-9fe9fe4074a4 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:18.193644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 869/1726] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:18.193886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a54597d-5ed0-4717-9057-83b39c2850a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:18.193886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a54597d-5ed0-4717-9057-83b39c2850a5 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:18.193886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a54597d-5ed0-4717-9057-83b39c2850a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:18.193886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a54597d-5ed0-4717-9057-83b39c2850a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:18.194156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a54597d-5ed0-4717-9057-83b39c2850a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:18.194156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a54597d-5ed0-4717-9057-83b39c2850a5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:18.196136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cecce74-4f35-4309-94e5-9fe9fe4074a4 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:18.224567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cecce74-4f35-4309-94e5-9fe9fe4074a4 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:18.233831 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 858/1727] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:18.236607 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a54597d-5ed0-4717-9057-83b39c2850a5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:18.243520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc9b0a-123d-4fe9-93e1-91a0c0d4ba4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:18.244200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc9b0a-123d-4fe9-93e1-91a0c0d4ba4d admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:18.244804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc9b0a-123d-4fe9-93e1-91a0c0d4ba4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:18.244804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc9b0a-123d-4fe9-93e1-91a0c0d4ba4d admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:18.247317 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc9b0a-123d-4fe9-93e1-91a0c0d4ba4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:18.249390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc9b0a-123d-4fe9-93e1-91a0c0d4ba4d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:18.322273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a54597d-5ed0-4717-9057-83b39c2850a5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:18.353749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66916c1f-bb16-4525-b25a-438592fb200f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:18.356887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66916c1f-bb16-4525-b25a-438592fb200f admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:18.362278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66916c1f-bb16-4525-b25a-438592fb200f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:18.362278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66916c1f-bb16-4525-b25a-438592fb200f admin admin] PATH_INFO: `/v3/projects/b772920647834cc39985c939084a4ce3/users/698637e4511642d8aa242002401cd35a/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:18.370500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 870/1728] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:18 2022] POST /identity/v3/projects => generated 367 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:18.372121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc9b0a-123d-4fe9-93e1-91a0c0d4ba4d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:18.404536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66916c1f-bb16-4525-b25a-438592fb200f admin admin] RBAC: Authorizing `identity:create_grant(project_id=b772920647834cc39985c939084a4ce3, user_id=698637e4511642d8aa242002401cd35a, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:18.412555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66916c1f-bb16-4525-b25a-438592fb200f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:18.478235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 859/1729] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:18 2022] PUT /identity/v3/projects/b772920647834cc39985c939084a4ce3/users/698637e4511642d8aa242002401cd35a/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:18.479236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66916c1f-bb16-4525-b25a-438592fb200f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:18.541911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc9b0a-123d-4fe9-93e1-91a0c0d4ba4d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:18.662593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66916c1f-bb16-4525-b25a-438592fb200f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:18.675398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fa9fd6-81ea-4bb3-ab18-7af9549e94d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:18.680945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fa9fd6-81ea-4bb3-ab18-7af9549e94d5 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:18.680945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fa9fd6-81ea-4bb3-ab18-7af9549e94d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:18.680945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fa9fd6-81ea-4bb3-ab18-7af9549e94d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:18.680945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fa9fd6-81ea-4bb3-ab18-7af9549e94d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:18.683391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fa9fd6-81ea-4bb3-ab18-7af9549e94d5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:18.725980 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 860/1730] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:18 2022] GET /identity/v3/roles => generated 1420 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:18.727099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fa9fd6-81ea-4bb3-ab18-7af9549e94d5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:18.752102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fa9fd6-81ea-4bb3-ab18-7af9549e94d5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:18.759216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b6b690-0f0c-4069-88da-43d15783b62d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:18.759928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b6b690-0f0c-4069-88da-43d15783b62d admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:18.760212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b6b690-0f0c-4069-88da-43d15783b62d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:18.760668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b6b690-0f0c-4069-88da-43d15783b62d admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:18.761224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b6b690-0f0c-4069-88da-43d15783b62d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:18.764396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95744629-0704-470f-a6d5-4f4be43b0706 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:18.765354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95744629-0704-470f-a6d5-4f4be43b0706 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:18.769387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95744629-0704-470f-a6d5-4f4be43b0706 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:18.769387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95744629-0704-470f-a6d5-4f4be43b0706 admin admin] PATH_INFO: `/v3/projects/b772920647834cc39985c939084a4ce3/users/698637e4511642d8aa242002401cd35a/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:18.772460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95744629-0704-470f-a6d5-4f4be43b0706 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b772920647834cc39985c939084a4ce3, user_id=698637e4511642d8aa242002401cd35a, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:18.780484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95744629-0704-470f-a6d5-4f4be43b0706 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:18.790953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b6b690-0f0c-4069-88da-43d15783b62d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:18.817387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 861/1731] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:18 2022] PUT /identity/v3/projects/b772920647834cc39985c939084a4ce3/users/698637e4511642d8aa242002401cd35a/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:18.821222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399035d6-2d54-4a65-b433-bf533166f120 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:18.821591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399035d6-2d54-4a65-b433-bf533166f120 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:18.821943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399035d6-2d54-4a65-b433-bf533166f120 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:18.876988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-399035d6-2d54-4a65-b433-bf533166f120 None None] MFA Rules not processed for user `698637e4511642d8aa242002401cd35a`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:18.880786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-399035d6-2d54-4a65-b433-bf533166f120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:19.059320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 862/1732] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:18 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:19.072062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399035d6-2d54-4a65-b433-bf533166f120 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:19.073750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-399035d6-2d54-4a65-b433-bf533166f120 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:19.114565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 871/1733] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:18 2022] POST /identity/v3/users => generated 284 bytes in 743 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:19.119686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b6b690-0f0c-4069-88da-43d15783b62d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:19.155444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b6b690-0f0c-4069-88da-43d15783b62d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:19.243808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399035d6-2d54-4a65-b433-bf533166f120 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:19.260114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e0644a-a037-4580-9999-9d4a34754f9c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:19.261465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e0644a-a037-4580-9999-9d4a34754f9c service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:19.267300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e0644a-a037-4580-9999-9d4a34754f9c service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:19.267737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e0644a-a037-4580-9999-9d4a34754f9c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:19.303602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e0644a-a037-4580-9999-9d4a34754f9c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:19.306184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e0644a-a037-4580-9999-9d4a34754f9c service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:19.335861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a84c61a-2ef8-43d8-80df-5eb56026a509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:19.336828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a84c61a-2ef8-43d8-80df-5eb56026a509 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:19.361190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a84c61a-2ef8-43d8-80df-5eb56026a509 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:19.361190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a84c61a-2ef8-43d8-80df-5eb56026a509 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:19.361190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a84c61a-2ef8-43d8-80df-5eb56026a509 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:19.409335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 863/1734] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:19 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 273 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:19.410541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a84c61a-2ef8-43d8-80df-5eb56026a509 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:19.410541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 872/1735] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:19 2022] GET /identity/v3/roles => generated 1420 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:19.411013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e0644a-a037-4580-9999-9d4a34754f9c service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:19.411196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e915145-b600-4dda-9b29-0b378db87470 None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:19.411196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e915145-b600-4dda-9b29-0b378db87470 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:19.411196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e915145-b600-4dda-9b29-0b378db87470 None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:19.411676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e0644a-a037-4580-9999-9d4a34754f9c service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:19.411772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 873/1736] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 13:05:19 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:19.411857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5054ed85-27a7-4cad-812f-e0bb132aa363 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:19.411857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5054ed85-27a7-4cad-812f-e0bb132aa363 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:19.411857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5054ed85-27a7-4cad-812f-e0bb132aa363 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:19.411857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5054ed85-27a7-4cad-812f-e0bb132aa363 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:19.412159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5054ed85-27a7-4cad-812f-e0bb132aa363 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:19.412159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5054ed85-27a7-4cad-812f-e0bb132aa363 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:19.438457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e915145-b600-4dda-9b29-0b378db87470 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:19.439533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 864/1737] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:19 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:19.443030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50df1267-802f-42dc-9117-41571e851691 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:19.443030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50df1267-802f-42dc-9117-41571e851691 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:19.443278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50df1267-802f-42dc-9117-41571e851691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:19.459619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e915145-b600-4dda-9b29-0b378db87470 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:19.470012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488befa2-bd77-435d-acb1-c6dc87a17313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:19.471711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488befa2-bd77-435d-acb1-c6dc87a17313 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:19.472158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488befa2-bd77-435d-acb1-c6dc87a17313 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:19.472644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488befa2-bd77-435d-acb1-c6dc87a17313 admin admin] PATH_INFO: `/v3/projects/eead5ae653c049a7a83139ca8151561a/users/8c512595c70b47e886be2034beaad0a2/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:19.484708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-50df1267-802f-42dc-9117-41571e851691 None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:19.487826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-50df1267-802f-42dc-9117-41571e851691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:19.512089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488befa2-bd77-435d-acb1-c6dc87a17313 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eead5ae653c049a7a83139ca8151561a, user_id=8c512595c70b47e886be2034beaad0a2, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:19.528066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488befa2-bd77-435d-acb1-c6dc87a17313 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:19.623284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 874/1738] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:19 2022] PUT /identity/v3/projects/eead5ae653c049a7a83139ca8151561a/users/8c512595c70b47e886be2034beaad0a2/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:19.633991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774fe98d-05d1-4203-b6d1-da1feeb3a409 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:19.634248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774fe98d-05d1-4203-b6d1-da1feeb3a409 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:19.649156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774fe98d-05d1-4203-b6d1-da1feeb3a409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:19.714130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 865/1739] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 13:05:19 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:19.771850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-774fe98d-05d1-4203-b6d1-da1feeb3a409 None None] MFA Rules not processed for user `8c512595c70b47e886be2034beaad0a2`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:19.774436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-774fe98d-05d1-4203-b6d1-da1feeb3a409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:19.778884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50df1267-802f-42dc-9117-41571e851691 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:19.780375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-50df1267-802f-42dc-9117-41571e851691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:19.846356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50df1267-802f-42dc-9117-41571e851691 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:20.206102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 875/1740] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:19 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 570 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:20.206102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774fe98d-05d1-4203-b6d1-da1feeb3a409 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:20.209772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-774fe98d-05d1-4203-b6d1-da1feeb3a409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:20.230260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff318f0b-9bb7-406f-92cf-c06ee80f3a70 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:20.231278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff318f0b-9bb7-406f-92cf-c06ee80f3a70 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:20.231567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff318f0b-9bb7-406f-92cf-c06ee80f3a70 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:20.233090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff318f0b-9bb7-406f-92cf-c06ee80f3a70 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:20.272889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff318f0b-9bb7-406f-92cf-c06ee80f3a70 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:20.285743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff318f0b-9bb7-406f-92cf-c06ee80f3a70 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:20.287280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:20.344296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:20.346715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 866/1741] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:19 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 569 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:20.349635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff318f0b-9bb7-406f-92cf-c06ee80f3a70 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:20.354224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ff318f0b-9bb7-406f-92cf-c06ee80f3a70 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:20.420792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774fe98d-05d1-4203-b6d1-da1feeb3a409 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:20.435776 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af51bb4-7005-403f-81a4-83cdc8905b2b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:20.438736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af51bb4-7005-403f-81a4-83cdc8905b2b service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:20.439253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af51bb4-7005-403f-81a4-83cdc8905b2b service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:20.439643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af51bb4-7005-403f-81a4-83cdc8905b2b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:20.447437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5af51bb4-7005-403f-81a4-83cdc8905b2b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:20.541654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff318f0b-9bb7-406f-92cf-c06ee80f3a70 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:20.567007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:20.572292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:20.574516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:20.575015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] PATH_INFO: `/v3/projects/1c071c7ca67f4cd788e1fc5b981e6494` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:20.585402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c071c7ca67f4cd788e1fc5b981e6494)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:20.589491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:20.601963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af51bb4-7005-403f-81a4-83cdc8905b2b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:20.601963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af51bb4-7005-403f-81a4-83cdc8905b2b service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:20.625104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 876/1742] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:20 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:20.626743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af51bb4-7005-403f-81a4-83cdc8905b2b service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:20.672352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af51bb4-7005-403f-81a4-83cdc8905b2b service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:20.865309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8f3406-707c-43f0-a286-63e26c8fde68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:20.865309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8f3406-707c-43f0-a286-63e26c8fde68 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:20.865309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8f3406-707c-43f0-a286-63e26c8fde68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:20.865309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8f3406-707c-43f0-a286-63e26c8fde68 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:20.866357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8f3406-707c-43f0-a286-63e26c8fde68 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:20.866357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8f3406-707c-43f0-a286-63e26c8fde68 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:21.053064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 877/1743] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:20 2022] POST /identity/v3/projects => generated 369 bytes in 427 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:21.057866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8f3406-707c-43f0-a286-63e26c8fde68 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:21.084693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c071c7ca67f4cd788e1fc5b981e6494', 'request_id': 'req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:21.248573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8f3406-707c-43f0-a286-63e26c8fde68 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:21.425335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9b2a52-bb66-439d-a2db-73a64326aa0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:21.426375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b2a52-bb66-439d-a2db-73a64326aa0f admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:21.426672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b2a52-bb66-439d-a2db-73a64326aa0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:21.427298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9b2a52-bb66-439d-a2db-73a64326aa0f admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:21.428443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9b2a52-bb66-439d-a2db-73a64326aa0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:21.431907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a9b2a52-bb66-439d-a2db-73a64326aa0f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:21.548609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c071c7ca67f4cd788e1fc5b981e6494', 'request_id': 'req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:21.567731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c071c7ca67f4cd788e1fc5b981e6494', 'request_id': 'req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:21.594424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] The token cache is being invalidate because project 1c071c7ca67f4cd788e1fc5b981e6494 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:21.595109 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:21.596186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-25eccd64-8c8e-4ccc-a34a-37ac79569d3f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:21.599031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 867/1744] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:20 2022] DELETE /identity/v3/projects/1c071c7ca67f4cd788e1fc5b981e6494 => generated 0 bytes in 1251 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:21.604694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713717ed-676d-4add-9a20-35a13e50082a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:21.605912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713717ed-676d-4add-9a20-35a13e50082a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:21.606362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713717ed-676d-4add-9a20-35a13e50082a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:21.657155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 878/1745] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:21 2022] POST /identity/v3/users => generated 291 bytes in 590 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:05:21.659244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9b2a52-bb66-439d-a2db-73a64326aa0f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:21.662046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0a9b2a52-bb66-439d-a2db-73a64326aa0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:21.671629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-713717ed-676d-4add-9a20-35a13e50082a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:21.672942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-713717ed-676d-4add-9a20-35a13e50082a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:21.689480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 868/1746] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:21.706133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d23b0-1dbf-4c4b-a6f6-7d7961c6dd73 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:21.706842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d23b0-1dbf-4c4b-a6f6-7d7961c6dd73 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:21.709514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d23b0-1dbf-4c4b-a6f6-7d7961c6dd73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:21.709873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a9b2a52-bb66-439d-a2db-73a64326aa0f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:21.720954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971d88b3-866a-4ad8-bbe0-bb361fa0b417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:21.721674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971d88b3-866a-4ad8-bbe0-bb361fa0b417 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:21.721962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971d88b3-866a-4ad8-bbe0-bb361fa0b417 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:21.722242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971d88b3-866a-4ad8-bbe0-bb361fa0b417 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:21.725481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971d88b3-866a-4ad8-bbe0-bb361fa0b417 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:21.727267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971d88b3-866a-4ad8-bbe0-bb361fa0b417 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:21.753750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 879/1747] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:21 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:21.760060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971d88b3-866a-4ad8-bbe0-bb361fa0b417 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:21.783694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3b0d23b0-1dbf-4c4b-a6f6-7d7961c6dd73 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:21.785925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3b0d23b0-1dbf-4c4b-a6f6-7d7961c6dd73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:21.808133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971d88b3-866a-4ad8-bbe0-bb361fa0b417 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:21.822765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df3ca1-8bf4-4498-b881-6e5bd23b28c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:21.828967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 869/1748] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:21.830064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df3ca1-8bf4-4498-b881-6e5bd23b28c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:21.830515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df3ca1-8bf4-4498-b881-6e5bd23b28c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:21.831474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df3ca1-8bf4-4498-b881-6e5bd23b28c6 admin admin] PATH_INFO: `/v3/projects/24a316a0450644838ca3da63ba6902c0/users/0c0da84145fe42a1b7f17460fe4cfc4d/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:21.850562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d23b0-1dbf-4c4b-a6f6-7d7961c6dd73 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:21.891973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99df3ca1-8bf4-4498-b881-6e5bd23b28c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24a316a0450644838ca3da63ba6902c0, user_id=0c0da84145fe42a1b7f17460fe4cfc4d, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:21.911069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99df3ca1-8bf4-4498-b881-6e5bd23b28c6 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:21.911455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d23b0-1dbf-4c4b-a6f6-7d7961c6dd73 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:21.929515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc2dfc4-7fe3-4609-876e-107cab66db6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:21.929515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc2dfc4-7fe3-4609-876e-107cab66db6d admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:21.929515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc2dfc4-7fe3-4609-876e-107cab66db6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:21.929515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc2dfc4-7fe3-4609-876e-107cab66db6d admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:21.929830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc2dfc4-7fe3-4609-876e-107cab66db6d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:21.929830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc2dfc4-7fe3-4609-876e-107cab66db6d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:21.931134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bc2dfc4-7fe3-4609-876e-107cab66db6d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:22.008121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 880/1749] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:21 2022] PUT /identity/v3/projects/24a316a0450644838ca3da63ba6902c0/users/0c0da84145fe42a1b7f17460fe4cfc4d/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:22.008870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 870/1750] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:22.013416 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df3ca1-8bf4-4498-b881-6e5bd23b28c6 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:22.013561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc2dfc4-7fe3-4609-876e-107cab66db6d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:22.044208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc2dfc4-7fe3-4609-876e-107cab66db6d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:22.153690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df3ca1-8bf4-4498-b881-6e5bd23b28c6 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:22.162732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff244f2-afc7-4d0a-82f3-a4be86df036c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:22.163429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff244f2-afc7-4d0a-82f3-a4be86df036c admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.163715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff244f2-afc7-4d0a-82f3-a4be86df036c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.163993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ff244f2-afc7-4d0a-82f3-a4be86df036c admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:22.164703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff244f2-afc7-4d0a-82f3-a4be86df036c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:22.166675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ff244f2-afc7-4d0a-82f3-a4be86df036c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:22.248244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11b1d2a-0625-4818-b907-7eadffb684f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:22.249749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11b1d2a-0625-4818-b907-7eadffb684f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.249749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11b1d2a-0625-4818-b907-7eadffb684f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.250053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11b1d2a-0625-4818-b907-7eadffb684f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:22.255765 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11b1d2a-0625-4818-b907-7eadffb684f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:22.257812 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11b1d2a-0625-4818-b907-7eadffb684f0 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:22.272950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 871/1751] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:22 2022] GET /identity/v3/roles => generated 1420 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:22.279754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11b1d2a-0625-4818-b907-7eadffb684f0 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:22.309122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11b1d2a-0625-4818-b907-7eadffb684f0 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:22.315574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb34fbf-ee1c-451f-ae8c-a885e8bd8026 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:22.321661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb34fbf-ee1c-451f-ae8c-a885e8bd8026 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.321661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb34fbf-ee1c-451f-ae8c-a885e8bd8026 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.321661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb34fbf-ee1c-451f-ae8c-a885e8bd8026 admin admin] PATH_INFO: `/v3/projects/24a316a0450644838ca3da63ba6902c0/users/0c0da84145fe42a1b7f17460fe4cfc4d/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:22.321661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb34fbf-ee1c-451f-ae8c-a885e8bd8026 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24a316a0450644838ca3da63ba6902c0, user_id=0c0da84145fe42a1b7f17460fe4cfc4d, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:22.325832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb34fbf-ee1c-451f-ae8c-a885e8bd8026 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:22.390077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 872/1752] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:22 2022] PUT /identity/v3/projects/24a316a0450644838ca3da63ba6902c0/users/0c0da84145fe42a1b7f17460fe4cfc4d/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:22.402222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8e8e6d-106c-4525-9bb7-0b8a31185d0e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.402532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8e8e6d-106c-4525-9bb7-0b8a31185d0e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.402945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf8e8e6d-106c-4525-9bb7-0b8a31185d0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:22.412412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 881/1753] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:22 2022] POST /identity/v3/projects => generated 383 bytes in 399 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:22.417891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff244f2-afc7-4d0a-82f3-a4be86df036c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:22.459415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-cf8e8e6d-106c-4525-9bb7-0b8a31185d0e None None] MFA Rules not processed for user `0c0da84145fe42a1b7f17460fe4cfc4d`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:22.471181 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-cf8e8e6d-106c-4525-9bb7-0b8a31185d0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:22.542276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ff244f2-afc7-4d0a-82f3-a4be86df036c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:22.549715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0f3208-c525-42f5-827a-ff47e79d5ab2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:22.550895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0f3208-c525-42f5-827a-ff47e79d5ab2 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.551347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0f3208-c525-42f5-827a-ff47e79d5ab2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.552039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0f3208-c525-42f5-827a-ff47e79d5ab2 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:22.553658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0f3208-c525-42f5-827a-ff47e79d5ab2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:22.556083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a0f3208-c525-42f5-827a-ff47e79d5ab2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:22.574058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 873/1754] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:22 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:22.575145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e1ffd3-cace-4930-82ca-1cabd5464728 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.575856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e1ffd3-cace-4930-82ca-1cabd5464728 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.588303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e1ffd3-cace-4930-82ca-1cabd5464728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:22.637106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-d5e1ffd3-cace-4930-82ca-1cabd5464728 None None] MFA Rules not processed for user `b1d1712ea523498f92f258383863b906`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:22.638421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d5e1ffd3-cace-4930-82ca-1cabd5464728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:22.717808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 874/1755] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:22 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:22.719515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e1ffd3-cace-4930-82ca-1cabd5464728 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:22.722895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d5e1ffd3-cace-4930-82ca-1cabd5464728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:22.801518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 882/1756] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:22 2022] POST /identity/v3/users => generated 292 bytes in 386 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:22.806211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0f3208-c525-42f5-827a-ff47e79d5ab2 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:22.810169 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e1ffd3-cace-4930-82ca-1cabd5464728 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:22.815313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd55d227-1896-4bda-9fbd-3797a22f1325 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:22.816102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd55d227-1896-4bda-9fbd-3797a22f1325 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.816399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd55d227-1896-4bda-9fbd-3797a22f1325 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.816809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd55d227-1896-4bda-9fbd-3797a22f1325 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:22.820609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0f3208-c525-42f5-827a-ff47e79d5ab2 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:22.828028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd55d227-1896-4bda-9fbd-3797a22f1325 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:22.832860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba4fe6d-e2d8-4d7f-a83f-b7b29b066afc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:22.832984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd55d227-1896-4bda-9fbd-3797a22f1325 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:22.833084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba4fe6d-e2d8-4d7f-a83f-b7b29b066afc admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.833084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba4fe6d-e2d8-4d7f-a83f-b7b29b066afc admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.833084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba4fe6d-e2d8-4d7f-a83f-b7b29b066afc admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:22.833084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba4fe6d-e2d8-4d7f-a83f-b7b29b066afc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:22.833084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba4fe6d-e2d8-4d7f-a83f-b7b29b066afc admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:22.847185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 875/1757] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:22 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:22.848506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd55d227-1896-4bda-9fbd-3797a22f1325 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:22.857968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 883/1758] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:22 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:22.862572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd55d227-1896-4bda-9fbd-3797a22f1325 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:22.873046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb79cec-e04c-4929-b4b2-7617e4e5c3d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:22.874412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb79cec-e04c-4929-b4b2-7617e4e5c3d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.874412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb79cec-e04c-4929-b4b2-7617e4e5c3d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.874412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb79cec-e04c-4929-b4b2-7617e4e5c3d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:22.875186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb79cec-e04c-4929-b4b2-7617e4e5c3d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:22.884476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb79cec-e04c-4929-b4b2-7617e4e5c3d4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:22.918563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 876/1759] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:22 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:22.927010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba4fe6d-e2d8-4d7f-a83f-b7b29b066afc admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:22.945365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba4fe6d-e2d8-4d7f-a83f-b7b29b066afc admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:22.959504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76916c3-0377-4ac4-8951-e4244a5d9cc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:22.960491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76916c3-0377-4ac4-8951-e4244a5d9cc8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:22.962267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76916c3-0377-4ac4-8951-e4244a5d9cc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:22.962552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76916c3-0377-4ac4-8951-e4244a5d9cc8 admin admin] PATH_INFO: `/v3/projects/dc1766a4cd254af785365d362242d607/users/216fa5869b984c28b63dfe185e7f793d/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:23.001577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76916c3-0377-4ac4-8951-e4244a5d9cc8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc1766a4cd254af785365d362242d607, user_id=216fa5869b984c28b63dfe185e7f793d, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:23.004515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb79cec-e04c-4929-b4b2-7617e4e5c3d4 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:23.004823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76916c3-0377-4ac4-8951-e4244a5d9cc8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:23.018674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb79cec-e04c-4929-b4b2-7617e4e5c3d4 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:23.030477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3cdb34-1fac-4359-ad02-06c30470b8d3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:23.031183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3cdb34-1fac-4359-ad02-06c30470b8d3 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:23.031415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3cdb34-1fac-4359-ad02-06c30470b8d3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:23.031693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3cdb34-1fac-4359-ad02-06c30470b8d3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:23.041509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ee3cdb34-1fac-4359-ad02-06c30470b8d3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:23.220779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3cdb34-1fac-4359-ad02-06c30470b8d3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:23.223221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3cdb34-1fac-4359-ad02-06c30470b8d3 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:23.239533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 884/1760] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:22 2022] PUT /identity/v3/projects/dc1766a4cd254af785365d362242d607/users/216fa5869b984c28b63dfe185e7f793d/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:23.259741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 877/1761] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:23 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:23.262314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee12490-d0f7-474b-b600-4e381d471417 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:23.262314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee12490-d0f7-474b-b600-4e381d471417 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:23.262314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee12490-d0f7-474b-b600-4e381d471417 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:23.307265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6ee12490-d0f7-474b-b600-4e381d471417 None None] MFA Rules not processed for user `216fa5869b984c28b63dfe185e7f793d`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:23.309665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6ee12490-d0f7-474b-b600-4e381d471417 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:23.505469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 885/1762] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:23.510314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3cdb34-1fac-4359-ad02-06c30470b8d3 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:23.531236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3cdb34-1fac-4359-ad02-06c30470b8d3 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:23.623561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1548774c-2a28-40b1-9438-c45c9204e56d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:23.627550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1548774c-2a28-40b1-9438-c45c9204e56d admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:23.629035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1548774c-2a28-40b1-9438-c45c9204e56d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:23.629437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1548774c-2a28-40b1-9438-c45c9204e56d admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:23.633619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1548774c-2a28-40b1-9438-c45c9204e56d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:23.635667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1548774c-2a28-40b1-9438-c45c9204e56d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:23.784135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 878/1763] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:23 2022] POST /identity/v3/projects => generated 383 bytes in 276 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:23.904702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee12490-d0f7-474b-b600-4e381d471417 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:23.919757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee12490-d0f7-474b-b600-4e381d471417 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:23.990503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf55966-7f62-4ceb-b692-07da10b0f569 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:23.991505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf55966-7f62-4ceb-b692-07da10b0f569 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:23.991948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf55966-7f62-4ceb-b692-07da10b0f569 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:23.992396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf55966-7f62-4ceb-b692-07da10b0f569 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:23.993296 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf55966-7f62-4ceb-b692-07da10b0f569 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:23.995910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf55966-7f62-4ceb-b692-07da10b0f569 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:24.240000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 886/1764] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:23 2022] POST /identity/v3/users => generated 298 bytes in 336 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:24.240983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1548774c-2a28-40b1-9438-c45c9204e56d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:24.273409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1548774c-2a28-40b1-9438-c45c9204e56d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:24.281640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f23a58-6206-4ea8-811b-bcb04da99d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:24.282434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f23a58-6206-4ea8-811b-bcb04da99d48 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:24.282791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f23a58-6206-4ea8-811b-bcb04da99d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:24.283072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f23a58-6206-4ea8-811b-bcb04da99d48 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:24.283828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f23a58-6206-4ea8-811b-bcb04da99d48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:24.285697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f23a58-6206-4ea8-811b-bcb04da99d48 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:24.303224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 879/1765] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:24 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:24.304737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf55966-7f62-4ceb-b692-07da10b0f569 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:24.315761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf55966-7f62-4ceb-b692-07da10b0f569 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:24.336350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfbbd31-435d-4b14-a714-b358cffcd5e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:24.337590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfbbd31-435d-4b14-a714-b358cffcd5e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:24.338027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfbbd31-435d-4b14-a714-b358cffcd5e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:24.339044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfbbd31-435d-4b14-a714-b358cffcd5e6 admin admin] PATH_INFO: `/v3/projects/446e33f922634d3db17683932f57fcfc/users/e99dbe43eaf0466d9dfb0ea0fd35265d/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:24.356307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfbbd31-435d-4b14-a714-b358cffcd5e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=446e33f922634d3db17683932f57fcfc, user_id=e99dbe43eaf0466d9dfb0ea0fd35265d, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:24.359318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cfbbd31-435d-4b14-a714-b358cffcd5e6 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:24.557118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f23a58-6206-4ea8-811b-bcb04da99d48 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:24.560834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 887/1766] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:24 2022] PUT /identity/v3/projects/446e33f922634d3db17683932f57fcfc/users/e99dbe43eaf0466d9dfb0ea0fd35265d/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:24.603172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f23a58-6206-4ea8-811b-bcb04da99d48 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:24.751431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b45b7291-d6d6-4164-aa39-5ef5e71bac58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:24.753147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b7291-d6d6-4164-aa39-5ef5e71bac58 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:24.756946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b7291-d6d6-4164-aa39-5ef5e71bac58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:24.767793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b45b7291-d6d6-4164-aa39-5ef5e71bac58 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:24.767793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b7291-d6d6-4164-aa39-5ef5e71bac58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:24.767793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b45b7291-d6d6-4164-aa39-5ef5e71bac58 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:24.798346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 880/1767] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:24 2022] GET /identity/v3/roles => generated 1420 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:24.798459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfbbd31-435d-4b14-a714-b358cffcd5e6 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:24.798459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:24.804386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfbbd31-435d-4b14-a714-b358cffcd5e6 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:24.836254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8caa752-c257-4735-8655-0e8e239ac8ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:24.837359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8caa752-c257-4735-8655-0e8e239ac8ef admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:24.837873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8caa752-c257-4735-8655-0e8e239ac8ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:24.838370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8caa752-c257-4735-8655-0e8e239ac8ef admin admin] PATH_INFO: `/v3/projects/446e33f922634d3db17683932f57fcfc/users/e99dbe43eaf0466d9dfb0ea0fd35265d/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:24.844848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:24.850612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8caa752-c257-4735-8655-0e8e239ac8ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=446e33f922634d3db17683932f57fcfc, user_id=e99dbe43eaf0466d9dfb0ea0fd35265d, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:24.853781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8caa752-c257-4735-8655-0e8e239ac8ef admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:24.889574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 888/1768] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:24 2022] PUT /identity/v3/projects/446e33f922634d3db17683932f57fcfc/users/e99dbe43eaf0466d9dfb0ea0fd35265d/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:24.901297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafe06a7-765b-4bb2-9e54-96f4050bdbbf None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:24.906225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafe06a7-765b-4bb2-9e54-96f4050bdbbf None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:24.906225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafe06a7-765b-4bb2-9e54-96f4050bdbbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:24.962567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-aafe06a7-765b-4bb2-9e54-96f4050bdbbf None None] MFA Rules not processed for user `e99dbe43eaf0466d9dfb0ea0fd35265d`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:24.966021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-aafe06a7-765b-4bb2-9e54-96f4050bdbbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:25.104739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 881/1769] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:24 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:25.170028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8caa752-c257-4735-8655-0e8e239ac8ef admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:25.185974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d8caa752-c257-4735-8655-0e8e239ac8ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:25.284114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8caa752-c257-4735-8655-0e8e239ac8ef admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:25.288937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c58dd84c-f62a-4269-ba83-67d36806d7f7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:25.289733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58dd84c-f62a-4269-ba83-67d36806d7f7 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:25.290171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58dd84c-f62a-4269-ba83-67d36806d7f7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:25.290450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c58dd84c-f62a-4269-ba83-67d36806d7f7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:25.301134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58dd84c-f62a-4269-ba83-67d36806d7f7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:25.302599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c58dd84c-f62a-4269-ba83-67d36806d7f7 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:25.314075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 889/1770] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:25 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:25.548474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafe06a7-765b-4bb2-9e54-96f4050bdbbf None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:25.564340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafe06a7-765b-4bb2-9e54-96f4050bdbbf None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:25.568642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95a7a2d-f996-40a0-874e-e1106443c188 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:25.569520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95a7a2d-f996-40a0-874e-e1106443c188 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:25.569994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95a7a2d-f996-40a0-874e-e1106443c188 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:25.570410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b95a7a2d-f996-40a0-874e-e1106443c188 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:25.658364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf8b062-fdf2-4b9e-85c7-471356f8d367 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:25.658888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf8b062-fdf2-4b9e-85c7-471356f8d367 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:25.658974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf8b062-fdf2-4b9e-85c7-471356f8d367 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:25.677578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-eaf8b062-fdf2-4b9e-85c7-471356f8d367 None None] MFA Rules not processed for user `9c8b2c27b69a4776ab6b7b7820da02d9`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:25.678800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-eaf8b062-fdf2-4b9e-85c7-471356f8d367 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:25.717135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95a7a2d-f996-40a0-874e-e1106443c188 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:25.719835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b95a7a2d-f996-40a0-874e-e1106443c188 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:25.736520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 882/1771] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:25 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:25.752889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 890/1772] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:25 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:26.374031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:28.276195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:28.333472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:28.910618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:29.161655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:29.213897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:30.231521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:30.491087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95a7a2d-f996-40a0-874e-e1106443c188 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:30.579518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b95a7a2d-f996-40a0-874e-e1106443c188 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:30.584323 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:30.585257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:30.585544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:30.585826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] PATH_INFO: `/v3/users/8c512595c70b47e886be2034beaad0a2` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:30.587122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c512595c70b47e886be2034beaad0a2)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:30.594860 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:30.840511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c512595c70b47e886be2034beaad0a2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:30.841367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c512595c70b47e886be2034beaad0a2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:30.842035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c512595c70b47e886be2034beaad0a2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:30.849172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c512595c70b47e886be2034beaad0a2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:30.910001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c512595c70b47e886be2034beaad0a2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:30.930720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 883/1773] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:30 2022] DELETE /identity/v3/users/8c512595c70b47e886be2034beaad0a2 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:30.935360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2448279-d690-4186-a891-ae27b4b84df2 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:30.935656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2448279-d690-4186-a891-ae27b4b84df2 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:30.935941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2448279-d690-4186-a891-ae27b4b84df2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:30.957332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e2448279-d690-4186-a891-ae27b4b84df2 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:30.958878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e2448279-d690-4186-a891-ae27b4b84df2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:30.991294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:31.054396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 891/1774] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:31.064493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:31.066252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:31.178728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fd04a-f1cb-4e39-b71e-1e0933a60f9e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:31.188153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda2b3cc-2a96-4447-aea3-c2a280f78ad5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:31.188961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda2b3cc-2a96-4447-aea3-c2a280f78ad5 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:31.189329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda2b3cc-2a96-4447-aea3-c2a280f78ad5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:31.189677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda2b3cc-2a96-4447-aea3-c2a280f78ad5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:31.216857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda2b3cc-2a96-4447-aea3-c2a280f78ad5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:31.218485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda2b3cc-2a96-4447-aea3-c2a280f78ad5 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:31.239951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 884/1775] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:31.750079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:31.844997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2448279-d690-4186-a891-ae27b4b84df2 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:31.846435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e2448279-d690-4186-a891-ae27b4b84df2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:31.862185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2448279-d690-4186-a891-ae27b4b84df2 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:31.889751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:31.890636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:31.891059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:31.891271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] PATH_INFO: `/v3/projects/eead5ae653c049a7a83139ca8151561a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:31.896804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eead5ae653c049a7a83139ca8151561a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:31.896804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:32.042444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eead5ae653c049a7a83139ca8151561a', 'request_id': 'req-bd06683e-34ca-4516-b235-7382bd40bf53'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.049351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eead5ae653c049a7a83139ca8151561a', 'request_id': 'req-bd06683e-34ca-4516-b235-7382bd40bf53'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.055500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eead5ae653c049a7a83139ca8151561a', 'request_id': 'req-bd06683e-34ca-4516-b235-7382bd40bf53'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.224714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] The token cache is being invalidate because project eead5ae653c049a7a83139ca8151561a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:32.225353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.226161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.231386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 892/1776] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:31 2022] DELETE /identity/v3/projects/eead5ae653c049a7a83139ca8151561a => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:32.232627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda2b3cc-2a96-4447-aea3-c2a280f78ad5 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:32.243562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda2b3cc-2a96-4447-aea3-c2a280f78ad5 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:32.341387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:32.342890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:32.343172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:32.343598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] PATH_INFO: `/v3/users/0c0da84145fe42a1b7f17460fe4cfc4d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:32.344966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c0da84145fe42a1b7f17460fe4cfc4d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:32.347102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:32.610169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:32.611780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:32.628374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd06683e-34ca-4516-b235-7382bd40bf53 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:32.634504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:32.635511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:32.635865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:32.636370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] PATH_INFO: `/v3/users/216fa5869b984c28b63dfe185e7f793d` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:32.638625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=216fa5869b984c28b63dfe185e7f793d)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:32.640996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:32.715080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c0da84145fe42a1b7f17460fe4cfc4d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.715923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c0da84145fe42a1b7f17460fe4cfc4d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.716623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c0da84145fe42a1b7f17460fe4cfc4d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.724749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c0da84145fe42a1b7f17460fe4cfc4d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.870000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cd55705c-d8fc-4df0-97b8-4a3c5d64e560 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c0da84145fe42a1b7f17460fe4cfc4d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:32.904878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 885/1777] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:32 2022] DELETE /identity/v3/users/0c0da84145fe42a1b7f17460fe4cfc4d => generated 0 bytes in 673 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:32.910090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3231d4b1-d3a7-4a78-828a-3a22b9027c8d None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:32.910448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3231d4b1-d3a7-4a78-828a-3a22b9027c8d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:32.910743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3231d4b1-d3a7-4a78-828a-3a22b9027c8d None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:32.915749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 886/1778] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 13:05:32 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:05:32.938812 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3cbc1d-f396-4022-93b6-2d4ff711915a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:32.939366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3cbc1d-f396-4022-93b6-2d4ff711915a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:32.939806 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3cbc1d-f396-4022-93b6-2d4ff711915a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:32.964062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ed3cbc1d-f396-4022-93b6-2d4ff711915a None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:32.965845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ed3cbc1d-f396-4022-93b6-2d4ff711915a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:33.010471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:33.015826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '216fa5869b984c28b63dfe185e7f793d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.018883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '216fa5869b984c28b63dfe185e7f793d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.047533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '216fa5869b984c28b63dfe185e7f793d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.091033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 887/1779] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 13:05:32 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:33.103239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3cbc1d-f396-4022-93b6-2d4ff711915a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:33.104737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ed3cbc1d-f396-4022-93b6-2d4ff711915a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:33.120196 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3cbc1d-f396-4022-93b6-2d4ff711915a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:33.126738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a143a977-af4b-42b2-85ec-5ec40bd822b2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:33.127406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a143a977-af4b-42b2-85ec-5ec40bd822b2 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:33.127691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a143a977-af4b-42b2-85ec-5ec40bd822b2 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:33.127972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a143a977-af4b-42b2-85ec-5ec40bd822b2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:33.137353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a143a977-af4b-42b2-85ec-5ec40bd822b2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:33.138776 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '216fa5869b984c28b63dfe185e7f793d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.139215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a143a977-af4b-42b2-85ec-5ec40bd822b2 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:33.139418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '216fa5869b984c28b63dfe185e7f793d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.150709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 888/1780] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:33 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:33.159197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 893/1781] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:32 2022] DELETE /identity/v3/users/216fa5869b984c28b63dfe185e7f793d => generated 0 bytes in 550 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:33.163778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5755cd98-ecf1-44e9-8091-f3f624b43180 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:33.164345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5755cd98-ecf1-44e9-8091-f3f624b43180 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:33.164713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5755cd98-ecf1-44e9-8091-f3f624b43180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:33.184804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5755cd98-ecf1-44e9-8091-f3f624b43180 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:33.186048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5755cd98-ecf1-44e9-8091-f3f624b43180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:33.269878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 889/1782] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:33.295690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:33.298652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:33.484076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c83247-95d4-4506-8cae-5f34406ebb4a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:33.499899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92bb3c9b-8efa-4d41-8856-ff67ff6fb911 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:33.500640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bb3c9b-8efa-4d41-8856-ff67ff6fb911 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:33.502569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bb3c9b-8efa-4d41-8856-ff67ff6fb911 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:33.506567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bb3c9b-8efa-4d41-8856-ff67ff6fb911 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:33.524103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5755cd98-ecf1-44e9-8091-f3f624b43180 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:33.526019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bb3c9b-8efa-4d41-8856-ff67ff6fb911 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:33.526192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5755cd98-ecf1-44e9-8091-f3f624b43180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:33.529922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92bb3c9b-8efa-4d41-8856-ff67ff6fb911 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:33.549325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 894/1783] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:33.551429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5755cd98-ecf1-44e9-8091-f3f624b43180 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:33.557106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:33.558062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:33.558396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:33.558677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] PATH_INFO: `/v3/projects/24a316a0450644838ca3da63ba6902c0` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:33.559908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] RBAC: Authorizing `identity:delete_project(project_id=24a316a0450644838ca3da63ba6902c0)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:33.564424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:33.770654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24a316a0450644838ca3da63ba6902c0', 'request_id': 'req-652d6044-93c6-4882-b82a-76509084d437'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.864253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24a316a0450644838ca3da63ba6902c0', 'request_id': 'req-652d6044-93c6-4882-b82a-76509084d437'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.871597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24a316a0450644838ca3da63ba6902c0', 'request_id': 'req-652d6044-93c6-4882-b82a-76509084d437'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.882338 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] The token cache is being invalidate because project 24a316a0450644838ca3da63ba6902c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:33.882653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.883325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-652d6044-93c6-4882-b82a-76509084d437 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:33.885011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 890/1784] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:33 2022] DELETE /identity/v3/projects/24a316a0450644838ca3da63ba6902c0 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:33.894911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50aa77b-490b-40bd-9657-6e0796f033fb None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:33.895862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50aa77b-490b-40bd-9657-6e0796f033fb None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:33.896533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50aa77b-490b-40bd-9657-6e0796f033fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:33.915652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c50aa77b-490b-40bd-9657-6e0796f033fb None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:33.917061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c50aa77b-490b-40bd-9657-6e0796f033fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:33.994386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 895/1785] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:34.002150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcb964d-1897-48e0-9c49-759e6cafbc23 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.002523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcb964d-1897-48e0-9c49-759e6cafbc23 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.002811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dcb964d-1897-48e0-9c49-759e6cafbc23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.030574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3dcb964d-1897-48e0-9c49-759e6cafbc23 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:34.030574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3dcb964d-1897-48e0-9c49-759e6cafbc23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:34.050275 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 891/1786] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:34.051289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50aa77b-490b-40bd-9657-6e0796f033fb None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:34.068630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50aa77b-490b-40bd-9657-6e0796f033fb None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:34.075362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cab58e7-0d9f-46bb-8d7b-dfd740570c65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:34.076101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cab58e7-0d9f-46bb-8d7b-dfd740570c65 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.076552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cab58e7-0d9f-46bb-8d7b-dfd740570c65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.077088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cab58e7-0d9f-46bb-8d7b-dfd740570c65 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.077992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cab58e7-0d9f-46bb-8d7b-dfd740570c65 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:34.078450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cab58e7-0d9f-46bb-8d7b-dfd740570c65 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:34.080298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cab58e7-0d9f-46bb-8d7b-dfd740570c65 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:34.095470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 896/1787] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:34.100109 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcb964d-1897-48e0-9c49-759e6cafbc23 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:34.111436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dcb964d-1897-48e0-9c49-759e6cafbc23 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:34.117474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:34.117859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51be94ac-be5c-43c0-87bb-a20d64c31f61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:34.118798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51be94ac-be5c-43c0-87bb-a20d64c31f61 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.119146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51be94ac-be5c-43c0-87bb-a20d64c31f61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.119361 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51be94ac-be5c-43c0-87bb-a20d64c31f61 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.120185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51be94ac-be5c-43c0-87bb-a20d64c31f61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:34.122704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51be94ac-be5c-43c0-87bb-a20d64c31f61 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:34.237975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 892/1788] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:34 2022] POST /identity/v3/projects => generated 387 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:05:34.242421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cab58e7-0d9f-46bb-8d7b-dfd740570c65 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:34.254315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cab58e7-0d9f-46bb-8d7b-dfd740570c65 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:34.347668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2537a03-2960-4399-a17a-b59453b5c7bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:34.348569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2537a03-2960-4399-a17a-b59453b5c7bd admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.348800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2537a03-2960-4399-a17a-b59453b5c7bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.349075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2537a03-2960-4399-a17a-b59453b5c7bd admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.350482 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2537a03-2960-4399-a17a-b59453b5c7bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:34.352583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2537a03-2960-4399-a17a-b59453b5c7bd admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:34.405222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51be94ac-be5c-43c0-87bb-a20d64c31f61 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:34.407581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-51be94ac-be5c-43c0-87bb-a20d64c31f61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:34.425499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 897/1789] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:34 2022] POST /identity/v3/users => generated 294 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:05:34.426227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51be94ac-be5c-43c0-87bb-a20d64c31f61 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:34.426711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2537a03-2960-4399-a17a-b59453b5c7bd admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:34.434785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:34.435940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.436321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.436633 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] PATH_INFO: `/v3/projects/dc1766a4cd254af785365d362242d607` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.441951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc1766a4cd254af785365d362242d607)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:34.442514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:34.447082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2537a03-2960-4399-a17a-b59453b5c7bd admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:34.453248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1899026c-794c-4825-b80a-b0cd2fce2357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:34.454014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1899026c-794c-4825-b80a-b0cd2fce2357 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.454287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1899026c-794c-4825-b80a-b0cd2fce2357 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.454569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1899026c-794c-4825-b80a-b0cd2fce2357 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.455316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1899026c-794c-4825-b80a-b0cd2fce2357 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:34.457030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1899026c-794c-4825-b80a-b0cd2fce2357 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:34.468050 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 898/1790] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:34 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:34.472311 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1899026c-794c-4825-b80a-b0cd2fce2357 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:34.483750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1899026c-794c-4825-b80a-b0cd2fce2357 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:34.489151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a79cd5a-3e0f-43d3-95bd-0123ef7b5244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:34.489888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a79cd5a-3e0f-43d3-95bd-0123ef7b5244 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.490952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a79cd5a-3e0f-43d3-95bd-0123ef7b5244 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.490952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a79cd5a-3e0f-43d3-95bd-0123ef7b5244 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.491204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a79cd5a-3e0f-43d3-95bd-0123ef7b5244 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:34.492999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a79cd5a-3e0f-43d3-95bd-0123ef7b5244 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:34.508855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 899/1791] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:34 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:34.509788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a79cd5a-3e0f-43d3-95bd-0123ef7b5244 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:34.521213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a79cd5a-3e0f-43d3-95bd-0123ef7b5244 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:34.526685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a156b7c-cdba-4487-9716-4b2c45d05336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:34.527585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a156b7c-cdba-4487-9716-4b2c45d05336 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.528459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a156b7c-cdba-4487-9716-4b2c45d05336 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.528751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a156b7c-cdba-4487-9716-4b2c45d05336 admin admin] PATH_INFO: `/v3/projects/ff3dc6c5da044b4ea7bff988ce5ad7c2/users/91bf7a1654494392bb4b9dbd6226edca/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.552444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a156b7c-cdba-4487-9716-4b2c45d05336 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff3dc6c5da044b4ea7bff988ce5ad7c2, user_id=91bf7a1654494392bb4b9dbd6226edca, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:34.553932 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a156b7c-cdba-4487-9716-4b2c45d05336 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:34.641254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 900/1792] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:34 2022] PUT /identity/v3/projects/ff3dc6c5da044b4ea7bff988ce5ad7c2/users/91bf7a1654494392bb4b9dbd6226edca/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:34.642302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a156b7c-cdba-4487-9716-4b2c45d05336 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:34.646602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc1766a4cd254af785365d362242d607', 'request_id': 'req-951c7915-95bf-4d90-979f-6b65c325a6dc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:34.650771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2a156b7c-cdba-4487-9716-4b2c45d05336 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:34.740301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc1766a4cd254af785365d362242d607', 'request_id': 'req-951c7915-95bf-4d90-979f-6b65c325a6dc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:34.753731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc1766a4cd254af785365d362242d607', 'request_id': 'req-951c7915-95bf-4d90-979f-6b65c325a6dc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:34.775648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] The token cache is being invalidate because project dc1766a4cd254af785365d362242d607 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:34.775945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:34.777256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-951c7915-95bf-4d90-979f-6b65c325a6dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:34.779144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 893/1793] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:34 2022] DELETE /identity/v3/projects/dc1766a4cd254af785365d362242d607 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:34.781085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835c1242-b352-456a-ba51-9f030c6ec5a3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.781404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835c1242-b352-456a-ba51-9f030c6ec5a3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.781780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835c1242-b352-456a-ba51-9f030c6ec5a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.816248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a156b7c-cdba-4487-9716-4b2c45d05336 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:34.830487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-835c1242-b352-456a-ba51-9f030c6ec5a3 None None] MFA Rules not processed for user `91bf7a1654494392bb4b9dbd6226edca`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:34.832354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-835c1242-b352-456a-ba51-9f030c6ec5a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:34.961552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:34.965661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:34.971319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:34.972005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] PATH_INFO: `/v3/users/dcdbde8f62d142949bd966d6e3089bff` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:34.977677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] RBAC: Authorizing `identity:delete_user(user_id=dcdbde8f62d142949bd966d6e3089bff)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:34.980087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:34.984476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:34.987493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 894/1794] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:34 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:34.988739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835c1242-b352-456a-ba51-9f030c6ec5a3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:34.992391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-835c1242-b352-456a-ba51-9f030c6ec5a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:35.022175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835c1242-b352-456a-ba51-9f030c6ec5a3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:35.034300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:35.035216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:35.035509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:35.035798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] PATH_INFO: `/v3/users/e99dbe43eaf0466d9dfb0ea0fd35265d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:35.037267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e99dbe43eaf0466d9dfb0ea0fd35265d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:35.039762 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:35.290428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dcdbde8f62d142949bd966d6e3089bff'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.291104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dcdbde8f62d142949bd966d6e3089bff'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.303856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dcdbde8f62d142949bd966d6e3089bff'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.375704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dcdbde8f62d142949bd966d6e3089bff'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.375704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dcdbde8f62d142949bd966d6e3089bff'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.392931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e99dbe43eaf0466d9dfb0ea0fd35265d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.394014 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e99dbe43eaf0466d9dfb0ea0fd35265d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.397584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e99dbe43eaf0466d9dfb0ea0fd35265d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.411062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e99dbe43eaf0466d9dfb0ea0fd35265d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.415677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 901/1795] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:34 2022] DELETE /identity/v3/users/dcdbde8f62d142949bd966d6e3089bff => generated 0 bytes in 773 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:35.416136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:35.417986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:35.497225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14a6954-7c95-4f11-bb53-5540532d1aae admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:35.502368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5525ae5e-5e9f-4779-abbb-5b93ece36c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:35.503154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5525ae5e-5e9f-4779-abbb-5b93ece36c89 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:35.503444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5525ae5e-5e9f-4779-abbb-5b93ece36c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:35.503911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5525ae5e-5e9f-4779-abbb-5b93ece36c89 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:35.504937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5525ae5e-5e9f-4779-abbb-5b93ece36c89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:35.507699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5525ae5e-5e9f-4779-abbb-5b93ece36c89 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:35.509893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e99dbe43eaf0466d9dfb0ea0fd35265d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.533748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 895/1796] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:34 2022] DELETE /identity/v3/users/e99dbe43eaf0466d9dfb0ea0fd35265d => generated 0 bytes in 545 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:35.534464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:35.535857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:35.615529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368f5606-75e9-4a9e-8fde-bba31ea6821e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:35.623487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:35.624754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:35.625723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:35.626195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] PATH_INFO: `/v3/users/d6617ab375314a9ab8e8276e81da0767` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:35.629002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6617ab375314a9ab8e8276e81da0767)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:35.631349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:35.671983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 902/1797] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:35 2022] POST /identity/v3/projects => generated 387 bytes in 256 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:35.673960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed7349e-775d-4c55-8816-e21b643f90d1 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:35.674453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed7349e-775d-4c55-8816-e21b643f90d1 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:35.674735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed7349e-775d-4c55-8816-e21b643f90d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:35.693620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4ed7349e-775d-4c55-8816-e21b643f90d1 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:35.694826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4ed7349e-775d-4c55-8816-e21b643f90d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:35.757385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 903/1798] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:35.758242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed7349e-775d-4c55-8816-e21b643f90d1 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:35.761794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4ed7349e-775d-4c55-8816-e21b643f90d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:35.777612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed7349e-775d-4c55-8816-e21b643f90d1 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:35.783389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f937f9bc-5f3a-49a2-bafb-6203dc6d3d8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:35.784801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f937f9bc-5f3a-49a2-bafb-6203dc6d3d8e admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:35.784801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f937f9bc-5f3a-49a2-bafb-6203dc6d3d8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:35.785136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f937f9bc-5f3a-49a2-bafb-6203dc6d3d8e admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:35.786028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f937f9bc-5f3a-49a2-bafb-6203dc6d3d8e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:35.789091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f937f9bc-5f3a-49a2-bafb-6203dc6d3d8e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:35.862761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 904/1799] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:35 2022] POST /identity/v3/users => generated 300 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:35.864509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f937f9bc-5f3a-49a2-bafb-6203dc6d3d8e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:35.866912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f937f9bc-5f3a-49a2-bafb-6203dc6d3d8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:35.877853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6617ab375314a9ab8e8276e81da0767'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.878538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6617ab375314a9ab8e8276e81da0767'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.879269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6617ab375314a9ab8e8276e81da0767'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.889644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6617ab375314a9ab8e8276e81da0767'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.950989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6617ab375314a9ab8e8276e81da0767'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:35.953817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:35.985002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 896/1800] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:35 2022] DELETE /identity/v3/users/d6617ab375314a9ab8e8276e81da0767 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:35.985894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:35.987937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:35.994771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f937f9bc-5f3a-49a2-bafb-6203dc6d3d8e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.078296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc48d149-8ef8-42c3-be13-c3ee73aedbaa admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.080798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:36.081548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.081833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.082154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.083579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b311dc6d-749a-40aa-8464-98bea7407df2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:36.084113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:36.084284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b311dc6d-749a-40aa-8464-98bea7407df2 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.084575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b311dc6d-749a-40aa-8464-98bea7407df2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.084893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b311dc6d-749a-40aa-8464-98bea7407df2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.088746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b311dc6d-749a-40aa-8464-98bea7407df2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:36.088746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b311dc6d-749a-40aa-8464-98bea7407df2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:36.096530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:36.099575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 897/1801] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:35 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:36.100218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:36.101799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a8969-d723-4df2-95e3-12d42020d560 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.106367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a8969-d723-4df2-95e3-12d42020d560 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.106367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201a8969-d723-4df2-95e3-12d42020d560 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.110986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 905/1802] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:36.112118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:36.114067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:36.122485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-201a8969-d723-4df2-95e3-12d42020d560 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:36.123616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-201a8969-d723-4df2-95e3-12d42020d560 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:36.133828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 898/1803] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:36.133828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201a8969-d723-4df2-95e3-12d42020d560 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:36.139101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e15f3ea-cefa-46cc-a1db-413ab7d8afb5 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.146861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201a8969-d723-4df2-95e3-12d42020d560 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.149062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:36.151422 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.151954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.152747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33845375-312c-44f9-bc40-c12a822ba4ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:36.153664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] PATH_INFO: `/v3/projects/446e33f922634d3db17683932f57fcfc` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.154353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33845375-312c-44f9-bc40-c12a822ba4ec admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.154754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33845375-312c-44f9-bc40-c12a822ba4ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.155035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33845375-312c-44f9-bc40-c12a822ba4ec admin admin] PATH_INFO: `/v3/projects/5fbaf674d16546caadc46447071a304f/users/5c631f48c8d04bd1aefd3d29c5533951/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.155896 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] RBAC: Authorizing `identity:delete_project(project_id=446e33f922634d3db17683932f57fcfc)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:36.158442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:36.190539 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33845375-312c-44f9-bc40-c12a822ba4ec admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fbaf674d16546caadc46447071a304f, user_id=5c631f48c8d04bd1aefd3d29c5533951, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:36.195767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33845375-312c-44f9-bc40-c12a822ba4ec admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:36.289276 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 899/1804] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:36 2022] PUT /identity/v3/projects/5fbaf674d16546caadc46447071a304f/users/5c631f48c8d04bd1aefd3d29c5533951/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:36.289964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33845375-312c-44f9-bc40-c12a822ba4ec admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:36.293497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-33845375-312c-44f9-bc40-c12a822ba4ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:36.329834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '446e33f922634d3db17683932f57fcfc', 'request_id': 'req-e716be97-3a3f-4925-b151-20688fb87f72'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:36.347309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '446e33f922634d3db17683932f57fcfc', 'request_id': 'req-e716be97-3a3f-4925-b151-20688fb87f72'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:36.356981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '446e33f922634d3db17683932f57fcfc', 'request_id': 'req-e716be97-3a3f-4925-b151-20688fb87f72'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:36.378208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33845375-312c-44f9-bc40-c12a822ba4ec admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.438745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fd5897e-1c58-4ee5-adf1-479b7a86d9fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:36.446035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd5897e-1c58-4ee5-adf1-479b7a86d9fd service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.446035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd5897e-1c58-4ee5-adf1-479b7a86d9fd service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.446035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fd5897e-1c58-4ee5-adf1-479b7a86d9fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.472339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] The token cache is being invalidate because project 446e33f922634d3db17683932f57fcfc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:36.472637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:36.473811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:36.475612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 906/1805] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:36 2022] DELETE /identity/v3/projects/446e33f922634d3db17683932f57fcfc => generated 0 bytes in 364 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:36.476403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:36.482331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:36.561434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:36.612325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd5897e-1c58-4ee5-adf1-479b7a86d9fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:36.615175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fd5897e-1c58-4ee5-adf1-479b7a86d9fd service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:36.636122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e716be97-3a3f-4925-b151-20688fb87f72 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.636867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 900/1806] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 345 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:36.636867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10ab5ed-a76f-43d4-b5b6-62848d2449a0 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.636867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10ab5ed-a76f-43d4-b5b6-62848d2449a0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.637364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c10ab5ed-a76f-43d4-b5b6-62848d2449a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.639668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57b7fa6-26e4-40c7-a1fb-2ec8077bd10a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:36.640405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57b7fa6-26e4-40c7-a1fb-2ec8077bd10a admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.640804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57b7fa6-26e4-40c7-a1fb-2ec8077bd10a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.641151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f57b7fa6-26e4-40c7-a1fb-2ec8077bd10a admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.641991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57b7fa6-26e4-40c7-a1fb-2ec8077bd10a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:36.643882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f57b7fa6-26e4-40c7-a1fb-2ec8077bd10a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:36.660462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-c10ab5ed-a76f-43d4-b5b6-62848d2449a0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:36.661945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c10ab5ed-a76f-43d4-b5b6-62848d2449a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:36.674544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 907/1807] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:36 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:36.677278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57b7fa6-26e4-40c7-a1fb-2ec8077bd10a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:36.687730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 901/1808] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:36.700244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e437ea8d-893a-4178-a4df-38b46527c865 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.700810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e437ea8d-893a-4178-a4df-38b46527c865 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.717173 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e437ea8d-893a-4178-a4df-38b46527c865 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.717745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f57b7fa6-26e4-40c7-a1fb-2ec8077bd10a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.733840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e437ea8d-893a-4178-a4df-38b46527c865 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:36.738879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df681b4c-d613-4af2-92b6-ba2cd5e1ce97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:36.740118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df681b4c-d613-4af2-92b6-ba2cd5e1ce97 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.740249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e437ea8d-893a-4178-a4df-38b46527c865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:36.740580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df681b4c-d613-4af2-92b6-ba2cd5e1ce97 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.741000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df681b4c-d613-4af2-92b6-ba2cd5e1ce97 admin admin] PATH_INFO: `/v3/projects/5fbaf674d16546caadc46447071a304f/users/5c631f48c8d04bd1aefd3d29c5533951/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.743297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df681b4c-d613-4af2-92b6-ba2cd5e1ce97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5fbaf674d16546caadc46447071a304f, user_id=5c631f48c8d04bd1aefd3d29c5533951, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:36.747645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df681b4c-d613-4af2-92b6-ba2cd5e1ce97 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:36.758636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 902/1809] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:36.762714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e437ea8d-893a-4178-a4df-38b46527c865 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:36.783505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 908/1810] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:36 2022] PUT /identity/v3/projects/5fbaf674d16546caadc46447071a304f/users/5c631f48c8d04bd1aefd3d29c5533951/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:36.786304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e437ea8d-893a-4178-a4df-38b46527c865 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.786440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af385d5b-e21d-4ba5-aca9-ca186f5cd654 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.786440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af385d5b-e21d-4ba5-aca9-ca186f5cd654 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.786440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af385d5b-e21d-4ba5-aca9-ca186f5cd654 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.841386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-af385d5b-e21d-4ba5-aca9-ca186f5cd654 None None] MFA Rules not processed for user `5c631f48c8d04bd1aefd3d29c5533951`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:36.842696 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-af385d5b-e21d-4ba5-aca9-ca186f5cd654 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:36.933677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:36.934551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.934983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.935380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.936972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:36.937445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:36.939210 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:36.959287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 903/1811] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:05:36.960596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:36.964469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:36.970316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 909/1812] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:36 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:36.970316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af385d5b-e21d-4ba5-aca9-ca186f5cd654 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:36.987492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910abb6f-0ed4-4a0f-a629-29e66e3c72c2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.990288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af385d5b-e21d-4ba5-aca9-ca186f5cd654 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:36.998666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:36.998666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:36.998666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:36.998666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] PATH_INFO: `/v3/users/fc9e0f8a8876497687fd7eab5795f31e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:36.998978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc9e0f8a8876497687fd7eab5795f31e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:37.001091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:37.025631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c24326-02a1-41f0-a948-40add2d0c7b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:37.026320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c24326-02a1-41f0-a948-40add2d0c7b6 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:37.026608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c24326-02a1-41f0-a948-40add2d0c7b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:37.032923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c24326-02a1-41f0-a948-40add2d0c7b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:37.032923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c24326-02a1-41f0-a948-40add2d0c7b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:37.035087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c24326-02a1-41f0-a948-40add2d0c7b6 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:37.170580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:37.177512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 910/1813] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:36 2022] POST /identity/v3/projects => generated 407 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:37.178569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c24326-02a1-41f0-a948-40add2d0c7b6 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:37.192258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-62c24326-02a1-41f0-a948-40add2d0c7b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:37.255100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c24326-02a1-41f0-a948-40add2d0c7b6 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:37.259280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c301232-d48b-463d-9722-7ceba14b7a05 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:37.264872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c301232-d48b-463d-9722-7ceba14b7a05 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:37.264872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c301232-d48b-463d-9722-7ceba14b7a05 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:37.264872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c301232-d48b-463d-9722-7ceba14b7a05 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:37.325529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c301232-d48b-463d-9722-7ceba14b7a05 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:37.327284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c301232-d48b-463d-9722-7ceba14b7a05 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:37.337908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 911/1814] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:37 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:37.338549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c301232-d48b-463d-9722-7ceba14b7a05 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:37.387685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc9e0f8a8876497687fd7eab5795f31e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.388384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc9e0f8a8876497687fd7eab5795f31e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.389095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc9e0f8a8876497687fd7eab5795f31e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.399893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc9e0f8a8876497687fd7eab5795f31e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.432167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c301232-d48b-463d-9722-7ceba14b7a05 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:37.447586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d1570-2ffd-4ac6-95b9-7762f8b522f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:37.447586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d1570-2ffd-4ac6-95b9-7762f8b522f7 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:37.447586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d1570-2ffd-4ac6-95b9-7762f8b522f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:37.447586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d1570-2ffd-4ac6-95b9-7762f8b522f7 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:37.448130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d1570-2ffd-4ac6-95b9-7762f8b522f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:37.448130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d1570-2ffd-4ac6-95b9-7762f8b522f7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:37.484392 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc9e0f8a8876497687fd7eab5795f31e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.511639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 904/1815] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:36 2022] DELETE /identity/v3/users/fc9e0f8a8876497687fd7eab5795f31e => generated 0 bytes in 549 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:37.511639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:37.511639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:37.566804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 912/1816] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:37 2022] POST /identity/v3/users => generated 304 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:37.567757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d1570-2ffd-4ac6-95b9-7762f8b522f7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:37.571771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-267d1570-2ffd-4ac6-95b9-7762f8b522f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:37.589186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d1570-2ffd-4ac6-95b9-7762f8b522f7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:37.628174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2d2917-8c8f-4e40-b828-22b4c2b1d8c3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:37.633247 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:37.637277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:37.637523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:37.637807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] PATH_INFO: `/v3/projects/ecf352f2fcaf489b8fed13a63da668d9` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:37.639274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ecf352f2fcaf489b8fed13a63da668d9)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:37.641423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:37.664685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:37.665607 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:37.665888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:37.666163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] PATH_INFO: `/v3/users/91bf7a1654494392bb4b9dbd6226edca` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:37.669245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=91bf7a1654494392bb4b9dbd6226edca)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:37.671307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:37.842528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ecf352f2fcaf489b8fed13a63da668d9', 'request_id': 'req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.843916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '91bf7a1654494392bb4b9dbd6226edca'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.844959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '91bf7a1654494392bb4b9dbd6226edca'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.863757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '91bf7a1654494392bb4b9dbd6226edca'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.921686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ecf352f2fcaf489b8fed13a63da668d9', 'request_id': 'req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.928928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ecf352f2fcaf489b8fed13a63da668d9', 'request_id': 'req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.947106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] The token cache is being invalidate because project ecf352f2fcaf489b8fed13a63da668d9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:37.948449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.948449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b5c73b7-71d5-45cd-a3aa-dbcc348beb45 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:37.949920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 905/1817] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:37 2022] DELETE /identity/v3/projects/ecf352f2fcaf489b8fed13a63da668d9 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:37.950992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559f6527-35ef-4d45-975e-751e0624c169 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:37.951263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559f6527-35ef-4d45-975e-751e0624c169 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:37.951546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-559f6527-35ef-4d45-975e-751e0624c169 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:37.969742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-559f6527-35ef-4d45-975e-751e0624c169 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:37.971005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-559f6527-35ef-4d45-975e-751e0624c169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.018440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '91bf7a1654494392bb4b9dbd6226edca'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.023825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '91bf7a1654494392bb4b9dbd6226edca'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.039710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 906/1818] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:38.041822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559f6527-35ef-4d45-975e-751e0624c169 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.046739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 913/1819] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:37 2022] DELETE /identity/v3/users/91bf7a1654494392bb4b9dbd6226edca => generated 0 bytes in 474 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:38.046940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-559f6527-35ef-4d45-975e-751e0624c169 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.047015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.047795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.058025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-559f6527-35ef-4d45-975e-751e0624c169 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:38.117120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fc4818c-ffaf-4a41-9d84-756ab33aba05 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:38.120466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a28f85-f770-48ff-9663-a5b623262914 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:38.121294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a28f85-f770-48ff-9663-a5b623262914 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.121585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a28f85-f770-48ff-9663-a5b623262914 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.121871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a28f85-f770-48ff-9663-a5b623262914 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.122060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:38.122728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a28f85-f770-48ff-9663-a5b623262914 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:38.122940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.123217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.123494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] PATH_INFO: `/v3/projects/82b81b58d92c4c7e999764af469c5683` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.124350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a28f85-f770-48ff-9663-a5b623262914 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:38.124692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82b81b58d92c4c7e999764af469c5683)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:38.126642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:38.134901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 907/1820] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:38 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:38.136304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636899bc-7585-48b9-8b53-5e3e44d32405 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.136600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636899bc-7585-48b9-8b53-5e3e44d32405 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.137103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636899bc-7585-48b9-8b53-5e3e44d32405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.157081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-636899bc-7585-48b9-8b53-5e3e44d32405 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:38.158380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-636899bc-7585-48b9-8b53-5e3e44d32405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.167161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 908/1821] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:38.169184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f51a17-6294-4496-8034-2fc62aff35be None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.169479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f51a17-6294-4496-8034-2fc62aff35be None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.169762 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f51a17-6294-4496-8034-2fc62aff35be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.190554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-81f51a17-6294-4496-8034-2fc62aff35be None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:38.191756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-81f51a17-6294-4496-8034-2fc62aff35be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.199508 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 909/1822] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:38.200414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f51a17-6294-4496-8034-2fc62aff35be None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.201631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-81f51a17-6294-4496-8034-2fc62aff35be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.258398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f51a17-6294-4496-8034-2fc62aff35be None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:38.269754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82b81b58d92c4c7e999764af469c5683', 'request_id': 'req-8d48388e-57e5-49b9-8c03-d173225bd213'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.275851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82b81b58d92c4c7e999764af469c5683', 'request_id': 'req-8d48388e-57e5-49b9-8c03-d173225bd213'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.282171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82b81b58d92c4c7e999764af469c5683', 'request_id': 'req-8d48388e-57e5-49b9-8c03-d173225bd213'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.313325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d417a55-af36-46e8-9d4d-fe6805390d36 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:38.313977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d417a55-af36-46e8-9d4d-fe6805390d36 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.314261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d417a55-af36-46e8-9d4d-fe6805390d36 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.314538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d417a55-af36-46e8-9d4d-fe6805390d36 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.371240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] The token cache is being invalidate because project 82b81b58d92c4c7e999764af469c5683 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:38.371742 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.372279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.373885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d417a55-af36-46e8-9d4d-fe6805390d36 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:38.374100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 914/1823] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:38 2022] DELETE /identity/v3/projects/82b81b58d92c4c7e999764af469c5683 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:38.374838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.375500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d417a55-af36-46e8-9d4d-fe6805390d36 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:38.376178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.385354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 910/1824] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:38.387156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b9611c-ba9d-4af8-a1a0-3cf2c66b7a0d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.387451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b9611c-ba9d-4af8-a1a0-3cf2c66b7a0d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.387733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b9611c-ba9d-4af8-a1a0-3cf2c66b7a0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.389423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d48388e-57e5-49b9-8c03-d173225bd213 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:38.394550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e821e0-b36b-4dcd-8951-c6c92d1df03c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:38.395218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e821e0-b36b-4dcd-8951-c6c92d1df03c admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.395506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e821e0-b36b-4dcd-8951-c6c92d1df03c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.395787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e821e0-b36b-4dcd-8951-c6c92d1df03c admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.396523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e821e0-b36b-4dcd-8951-c6c92d1df03c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:38.398274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e821e0-b36b-4dcd-8951-c6c92d1df03c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:38.407370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-36b9611c-ba9d-4af8-a1a0-3cf2c66b7a0d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:38.408684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-36b9611c-ba9d-4af8-a1a0-3cf2c66b7a0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.413501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 915/1825] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:38 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:38.413501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e821e0-b36b-4dcd-8951-c6c92d1df03c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.414846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d5e821e0-b36b-4dcd-8951-c6c92d1df03c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.418856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 911/1826] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:38.427285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b9611c-ba9d-4af8-a1a0-3cf2c66b7a0d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.427285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-36b9611c-ba9d-4af8-a1a0-3cf2c66b7a0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.427474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e821e0-b36b-4dcd-8951-c6c92d1df03c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:38.431842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8b43f4-4bf8-49b1-a912-d626ed23cbab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:38.432265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b43f4-4bf8-49b1-a912-d626ed23cbab service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.432596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b43f4-4bf8-49b1-a912-d626ed23cbab service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.433026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d8b43f4-4bf8-49b1-a912-d626ed23cbab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.435091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9d8b43f4-4bf8-49b1-a912-d626ed23cbab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.435255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b9611c-ba9d-4af8-a1a0-3cf2c66b7a0d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:38.440064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:38.441491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.441617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.441826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] PATH_INFO: `/v3/users/d138f9e1a64f4b298843c1dc9cf3fba1` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.443202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d138f9e1a64f4b298843c1dc9cf3fba1)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:38.445995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:38.449773 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8b43f4-4bf8-49b1-a912-d626ed23cbab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:38.452688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d8b43f4-4bf8-49b1-a912-d626ed23cbab service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:38.469384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 916/1827] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:38.470218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8b43f4-4bf8-49b1-a912-d626ed23cbab service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.480332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d8b43f4-4bf8-49b1-a912-d626ed23cbab service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:38.485078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec7d610-2c55-46c2-b8c2-7b87cb08174e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:38.486086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec7d610-2c55-46c2-b8c2-7b87cb08174e admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.486383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec7d610-2c55-46c2-b8c2-7b87cb08174e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.486663 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec7d610-2c55-46c2-b8c2-7b87cb08174e admin admin] PATH_INFO: `/v3/projects/f000c174b9184feab8726af9f46ac8c3/users/53d8c34a5e14484eb0366b0f6de4ebcd/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.502089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec7d610-2c55-46c2-b8c2-7b87cb08174e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f000c174b9184feab8726af9f46ac8c3, user_id=53d8c34a5e14484eb0366b0f6de4ebcd, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:38.504828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec7d610-2c55-46c2-b8c2-7b87cb08174e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:38.618652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 917/1828] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:38 2022] PUT /identity/v3/projects/f000c174b9184feab8726af9f46ac8c3/users/53d8c34a5e14484eb0366b0f6de4ebcd/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:38.619563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec7d610-2c55-46c2-b8c2-7b87cb08174e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.694709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec7d610-2c55-46c2-b8c2-7b87cb08174e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:38.701004 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67eb89a2-b4a6-47f4-a031-2d5dc1c7a254 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:38.702390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eb89a2-b4a6-47f4-a031-2d5dc1c7a254 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.702751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eb89a2-b4a6-47f4-a031-2d5dc1c7a254 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.703105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67eb89a2-b4a6-47f4-a031-2d5dc1c7a254 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.703920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67eb89a2-b4a6-47f4-a031-2d5dc1c7a254 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:38.704299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67eb89a2-b4a6-47f4-a031-2d5dc1c7a254 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:38.706116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67eb89a2-b4a6-47f4-a031-2d5dc1c7a254 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:38.736068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd138f9e1a64f4b298843c1dc9cf3fba1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.736990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd138f9e1a64f4b298843c1dc9cf3fba1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.739145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd138f9e1a64f4b298843c1dc9cf3fba1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.743230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 918/1829] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:38.744684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53656d-b585-45cf-8912-8ace47021e31 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.745096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53656d-b585-45cf-8912-8ace47021e31 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.746348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53656d-b585-45cf-8912-8ace47021e31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.766285 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd138f9e1a64f4b298843c1dc9cf3fba1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.791039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-cc53656d-b585-45cf-8912-8ace47021e31 None None] MFA Rules not processed for user `53d8c34a5e14484eb0366b0f6de4ebcd`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:38.792463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-cc53656d-b585-45cf-8912-8ace47021e31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.860920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 919/1830] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:38 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:38.862069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53656d-b585-45cf-8912-8ace47021e31 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.864537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-cc53656d-b585-45cf-8912-8ace47021e31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:38.870241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd138f9e1a64f4b298843c1dc9cf3fba1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:38.882901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53656d-b585-45cf-8912-8ace47021e31 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:38.889064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a62aca-fe04-4d30-8508-7c320390b2fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:38.889812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a62aca-fe04-4d30-8508-7c320390b2fc admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:38.890025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a62aca-fe04-4d30-8508-7c320390b2fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:38.890284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a62aca-fe04-4d30-8508-7c320390b2fc admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:38.891620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a62aca-fe04-4d30-8508-7c320390b2fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:38.893605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a62aca-fe04-4d30-8508-7c320390b2fc admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:38.913217 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 912/1831] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:38 2022] DELETE /identity/v3/users/d138f9e1a64f4b298843c1dc9cf3fba1 => generated 0 bytes in 493 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:38.914255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:38.918853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:39.032979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 920/1832] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:38 2022] POST /identity/v3/projects => generated 375 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:39.034092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a62aca-fe04-4d30-8508-7c320390b2fc admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:39.035941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c5a62aca-fe04-4d30-8508-7c320390b2fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:39.149776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4326edb-2405-43cc-aab2-040279455cd3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:39.154042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a62aca-fe04-4d30-8508-7c320390b2fc admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:39.155898 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85562ff6-21f9-4ede-a12c-8ec55fb4ece3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:39.156621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85562ff6-21f9-4ede-a12c-8ec55fb4ece3 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:39.156964 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85562ff6-21f9-4ede-a12c-8ec55fb4ece3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:39.157245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85562ff6-21f9-4ede-a12c-8ec55fb4ece3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:39.158017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85562ff6-21f9-4ede-a12c-8ec55fb4ece3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:39.159454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:39.159997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85562ff6-21f9-4ede-a12c-8ec55fb4ece3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:39.160457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:39.160875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:39.161259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] PATH_INFO: `/v3/projects/8f784f9f33f34372888b4575b5b42771` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:39.162521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f784f9f33f34372888b4575b5b42771)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:39.164697 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:39.293627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 913/1833] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:38 2022] POST /identity/v3/projects => generated 409 bytes in 380 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:39.295371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85562ff6-21f9-4ede-a12c-8ec55fb4ece3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:39.311708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f784f9f33f34372888b4575b5b42771', 'request_id': 'req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.320071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f784f9f33f34372888b4575b5b42771', 'request_id': 'req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.332748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f784f9f33f34372888b4575b5b42771', 'request_id': 'req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.379401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85562ff6-21f9-4ede-a12c-8ec55fb4ece3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:39.384163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f807e7-d7c9-4ff8-91f3-e4a694821fad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:39.384861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f807e7-d7c9-4ff8-91f3-e4a694821fad admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:39.385143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f807e7-d7c9-4ff8-91f3-e4a694821fad admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:39.385414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11f807e7-d7c9-4ff8-91f3-e4a694821fad admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:39.386676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f807e7-d7c9-4ff8-91f3-e4a694821fad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:39.388845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11f807e7-d7c9-4ff8-91f3-e4a694821fad admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:39.428592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] The token cache is being invalidate because project 8f784f9f33f34372888b4575b5b42771 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:39.429198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.429687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.431303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 921/1834] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:39 2022] DELETE /identity/v3/projects/8f784f9f33f34372888b4575b5b42771 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:39.432072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:39.434022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:39.490013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=93,keystone:INSERT=1,keystone:DELETE=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:39.526348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 914/1835] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:39 2022] POST /identity/v3/users => generated 288 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:39.526348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f807e7-d7c9-4ff8-91f3-e4a694821fad admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:39.526348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-11f807e7-d7c9-4ff8-91f3-e4a694821fad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:39.533024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-900a4d59-d9cb-4495-bbe3-eadfa1844e0f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:39.539728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:39.540966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:39.541175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11f807e7-d7c9-4ff8-91f3-e4a694821fad admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:39.541280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:39.541531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] PATH_INFO: `/v3/projects/ff3dc6c5da044b4ea7bff988ce5ad7c2` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:39.543018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff3dc6c5da044b4ea7bff988ce5ad7c2)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:39.545113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:39.547224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01dc8f5-64be-4bf9-a6a3-b26fe93954e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:39.548019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01dc8f5-64be-4bf9-a6a3-b26fe93954e6 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:39.548207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01dc8f5-64be-4bf9-a6a3-b26fe93954e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:39.548617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01dc8f5-64be-4bf9-a6a3-b26fe93954e6 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:39.549180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01dc8f5-64be-4bf9-a6a3-b26fe93954e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:39.551397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d01dc8f5-64be-4bf9-a6a3-b26fe93954e6 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:39.649150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 915/1836] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:39 2022] POST /identity/v3/users => generated 311 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:39.649150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01dc8f5-64be-4bf9-a6a3-b26fe93954e6 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:39.651032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d01dc8f5-64be-4bf9-a6a3-b26fe93954e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:39.665066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01dc8f5-64be-4bf9-a6a3-b26fe93954e6 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:39.666291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff3dc6c5da044b4ea7bff988ce5ad7c2', 'request_id': 'req-6abe4d20-71f4-4784-aa99-2a51a647d77f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.673376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff3dc6c5da044b4ea7bff988ce5ad7c2', 'request_id': 'req-6abe4d20-71f4-4784-aa99-2a51a647d77f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.679216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff3dc6c5da044b4ea7bff988ce5ad7c2', 'request_id': 'req-6abe4d20-71f4-4784-aa99-2a51a647d77f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.728493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:39.729410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:39.730222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:39.730500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] PATH_INFO: `/v3/users/698637e4511642d8aa242002401cd35a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:39.731776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] RBAC: Authorizing `identity:delete_user(user_id=698637e4511642d8aa242002401cd35a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:39.733760 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:39.766242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] The token cache is being invalidate because project ff3dc6c5da044b4ea7bff988ce5ad7c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:39.772333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.772333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.772333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 922/1837] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:39 2022] DELETE /identity/v3/projects/ff3dc6c5da044b4ea7bff988ce5ad7c2 => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:39.772333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:39.772333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:39.787933 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abe4d20-71f4-4784-aa99-2a51a647d77f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:39.793187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3bad42-1470-4592-a429-66ddb4d1526c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:39.798160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3bad42-1470-4592-a429-66ddb4d1526c admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:39.798160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3bad42-1470-4592-a429-66ddb4d1526c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:39.798160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3bad42-1470-4592-a429-66ddb4d1526c admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:39.798160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3bad42-1470-4592-a429-66ddb4d1526c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:39.798160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3bad42-1470-4592-a429-66ddb4d1526c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:39.808955 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 923/1838] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:39 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:39.811474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3bad42-1470-4592-a429-66ddb4d1526c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:39.813481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6d3bad42-1470-4592-a429-66ddb4d1526c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:39.831760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3bad42-1470-4592-a429-66ddb4d1526c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:39.837042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f9e66-be2f-45d4-8afc-a30b0317c8e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:39.837728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f9e66-be2f-45d4-8afc-a30b0317c8e5 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:39.838017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f9e66-be2f-45d4-8afc-a30b0317c8e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:39.838485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0f9e66-be2f-45d4-8afc-a30b0317c8e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:39.839314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0f9e66-be2f-45d4-8afc-a30b0317c8e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:39.840967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0f9e66-be2f-45d4-8afc-a30b0317c8e5 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:39.851383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 924/1839] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:39 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:39.852329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f9e66-be2f-45d4-8afc-a30b0317c8e5 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:39.855601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-cf0f9e66-be2f-45d4-8afc-a30b0317c8e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:39.867448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0f9e66-be2f-45d4-8afc-a30b0317c8e5 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:39.872887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:39.874114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:39.874283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:39.874523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] PATH_INFO: `/v3/users/5c631f48c8d04bd1aefd3d29c5533951` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:39.875954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c631f48c8d04bd1aefd3d29c5533951)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:39.878728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:39.920721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '698637e4511642d8aa242002401cd35a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.921439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '698637e4511642d8aa242002401cd35a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.922067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '698637e4511642d8aa242002401cd35a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:39.931085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '698637e4511642d8aa242002401cd35a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.019072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '698637e4511642d8aa242002401cd35a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.046030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 916/1840] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:39 2022] DELETE /identity/v3/users/698637e4511642d8aa242002401cd35a => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:40.047273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:40.048887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:40.123913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901992e5-1962-4e32-bb45-1b77fdc4109e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:40.129506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:40.130470 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:40.130833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:40.131292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] PATH_INFO: `/v3/projects/604f78c1c14a4bdba20916e82b772038` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:40.132570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] RBAC: Authorizing `identity:delete_project(project_id=604f78c1c14a4bdba20916e82b772038)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:40.134614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:40.166610 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c631f48c8d04bd1aefd3d29c5533951'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.167920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c631f48c8d04bd1aefd3d29c5533951'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.175025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c631f48c8d04bd1aefd3d29c5533951'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.332944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '604f78c1c14a4bdba20916e82b772038', 'request_id': 'req-ca2011d9-433d-4fee-8e92-1d715557de0b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.343366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c631f48c8d04bd1aefd3d29c5533951'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.343366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:40.343366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c631f48c8d04bd1aefd3d29c5533951'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.361891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 925/1841] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:39 2022] DELETE /identity/v3/users/5c631f48c8d04bd1aefd3d29c5533951 => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:40.362740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:40.364397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:40.441299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c05ea7d-807d-490a-9e51-169a8e38bf4f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:40.447094 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '604f78c1c14a4bdba20916e82b772038', 'request_id': 'req-ca2011d9-433d-4fee-8e92-1d715557de0b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.450280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4615e69d-1cd4-4682-bee2-89b662af2b22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:40.450975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4615e69d-1cd4-4682-bee2-89b662af2b22 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:40.451318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4615e69d-1cd4-4682-bee2-89b662af2b22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:40.451605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4615e69d-1cd4-4682-bee2-89b662af2b22 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:40.453069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4615e69d-1cd4-4682-bee2-89b662af2b22 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:40.454873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4615e69d-1cd4-4682-bee2-89b662af2b22 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:40.458016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '604f78c1c14a4bdba20916e82b772038', 'request_id': 'req-ca2011d9-433d-4fee-8e92-1d715557de0b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.470887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 926/1842] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:40 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:40.470887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4615e69d-1cd4-4682-bee2-89b662af2b22 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:40.472385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4615e69d-1cd4-4682-bee2-89b662af2b22 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:40.486821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4615e69d-1cd4-4682-bee2-89b662af2b22 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:40.492465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c3e70f-2d06-4210-8932-c4b0f680c69e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:40.493398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3e70f-2d06-4210-8932-c4b0f680c69e admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:40.493681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3e70f-2d06-4210-8932-c4b0f680c69e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:40.494047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3e70f-2d06-4210-8932-c4b0f680c69e admin admin] PATH_INFO: `/v3/projects/5280bec0e015484aa1bbb7092acc945e/users/5fd21861e41b4f43b74bacd53757f628/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:40.494828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] The token cache is being invalidate because project 604f78c1c14a4bdba20916e82b772038 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:40.495124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.497193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:40.503004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 917/1843] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:40 2022] DELETE /identity/v3/projects/604f78c1c14a4bdba20916e82b772038 => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:40.503645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:40.506244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:40.513563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3e70f-2d06-4210-8932-c4b0f680c69e admin admin] RBAC: Authorizing `identity:create_grant(project_id=5280bec0e015484aa1bbb7092acc945e, user_id=5fd21861e41b4f43b74bacd53757f628, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:40.516334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3e70f-2d06-4210-8932-c4b0f680c69e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:40.600568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca2011d9-433d-4fee-8e92-1d715557de0b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:40.605446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c053fc5c-0833-4854-a029-9f8e5da39937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:40.606349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c053fc5c-0833-4854-a029-9f8e5da39937 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:40.606634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c053fc5c-0833-4854-a029-9f8e5da39937 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:40.606912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c053fc5c-0833-4854-a029-9f8e5da39937 admin admin] PATH_INFO: `/v3/projects/7664faa3fbfe40deaf65d87e26cccab6/users/dd015e885aec411cb4eef64f5934d68d/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:40.621721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 927/1844] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:40 2022] PUT /identity/v3/projects/5280bec0e015484aa1bbb7092acc945e/users/5fd21861e41b4f43b74bacd53757f628/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:40.622992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c053fc5c-0833-4854-a029-9f8e5da39937 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7664faa3fbfe40deaf65d87e26cccab6, user_id=dd015e885aec411cb4eef64f5934d68d, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:40.623100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97324906-ae8d-4c2f-9117-10780051690f None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:40.623345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97324906-ae8d-4c2f-9117-10780051690f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:40.623625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97324906-ae8d-4c2f-9117-10780051690f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:40.626530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c053fc5c-0833-4854-a029-9f8e5da39937 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:40.642215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-97324906-ae8d-4c2f-9117-10780051690f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:40.643536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-97324906-ae8d-4c2f-9117-10780051690f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:40.725670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 928/1846] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:40.726715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97324906-ae8d-4c2f-9117-10780051690f None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:40.727814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-97324906-ae8d-4c2f-9117-10780051690f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:40.734353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 918/1846] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:40 2022] PUT /identity/v3/projects/7664faa3fbfe40deaf65d87e26cccab6/users/dd015e885aec411cb4eef64f5934d68d/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:40.742250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c053fc5c-0833-4854-a029-9f8e5da39937 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:40.746500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97324906-ae8d-4c2f-9117-10780051690f None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:40.750974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c053fc5c-0833-4854-a029-9f8e5da39937 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:40.829546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c983d1-b86f-4dad-b103-3f2b0e958193 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:40.830295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c983d1-b86f-4dad-b103-3f2b0e958193 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:40.830658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c983d1-b86f-4dad-b103-3f2b0e958193 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:40.831015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c983d1-b86f-4dad-b103-3f2b0e958193 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:40.831826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c983d1-b86f-4dad-b103-3f2b0e958193 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:40.833544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c983d1-b86f-4dad-b103-3f2b0e958193 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:40.834445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c053fc5c-0833-4854-a029-9f8e5da39937 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:40.840938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:40.841807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:40.842092 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:40.842371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] PATH_INFO: `/v3/projects/b772920647834cc39985c939084a4ce3` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:40.843662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b772920647834cc39985c939084a4ce3)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:40.845708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:40.846253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 929/1847] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:40 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:40.847696 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e20dded-0d04-4b03-8420-e837b73c6f4c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:40.847990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e20dded-0d04-4b03-8420-e837b73c6f4c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:40.848268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e20dded-0d04-4b03-8420-e837b73c6f4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:40.875852 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-9e20dded-0d04-4b03-8420-e837b73c6f4c None None] MFA Rules not processed for user `dd015e885aec411cb4eef64f5934d68d`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:40.877022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9e20dded-0d04-4b03-8420-e837b73c6f4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:40.943236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 930/1848] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:40 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:40.944793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196765b-1c79-401c-ae47-d52d07a9aff9 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:40.945077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196765b-1c79-401c-ae47-d52d07a9aff9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:40.945355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1196765b-1c79-401c-ae47-d52d07a9aff9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:40.965117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-1196765b-1c79-401c-ae47-d52d07a9aff9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:40.966434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1196765b-1c79-401c-ae47-d52d07a9aff9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:40.976644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 931/1849] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:40.980897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196765b-1c79-401c-ae47-d52d07a9aff9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:40.994904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1196765b-1c79-401c-ae47-d52d07a9aff9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.003991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b772920647834cc39985c939084a4ce3', 'request_id': 'req-bd37ac2e-4b7b-4485-b99c-2840cedbb057'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.048865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054c8b60-1a45-40ba-8678-4b2dce054f42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.049758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054c8b60-1a45-40ba-8678-4b2dce054f42 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.050044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054c8b60-1a45-40ba-8678-4b2dce054f42 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.050319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054c8b60-1a45-40ba-8678-4b2dce054f42 admin admin] PATH_INFO: `/v3/projects/5280bec0e015484aa1bbb7092acc945e/users/5fd21861e41b4f43b74bacd53757f628/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.052267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054c8b60-1a45-40ba-8678-4b2dce054f42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5280bec0e015484aa1bbb7092acc945e, user_id=5fd21861e41b4f43b74bacd53757f628, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.054940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054c8b60-1a45-40ba-8678-4b2dce054f42 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.065641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 932/1850] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:40 2022] PUT /identity/v3/projects/5280bec0e015484aa1bbb7092acc945e/users/5fd21861e41b4f43b74bacd53757f628/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:41.066406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054c8b60-1a45-40ba-8678-4b2dce054f42 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.069086 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b772920647834cc39985c939084a4ce3', 'request_id': 'req-bd37ac2e-4b7b-4485-b99c-2840cedbb057'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.076493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b772920647834cc39985c939084a4ce3', 'request_id': 'req-bd37ac2e-4b7b-4485-b99c-2840cedbb057'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.087244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] The token cache is being invalidate because project b772920647834cc39985c939084a4ce3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:41.087522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.088194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.089866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 919/1851] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:40 2022] DELETE /identity/v3/projects/b772920647834cc39985c939084a4ce3 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:41.090929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.092701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.104816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd37ac2e-4b7b-4485-b99c-2840cedbb057 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.127854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054c8b60-1a45-40ba-8678-4b2dce054f42 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.132645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f949349-900f-4fed-91a9-924d71ef8b4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.133318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f949349-900f-4fed-91a9-924d71ef8b4d admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.133601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f949349-900f-4fed-91a9-924d71ef8b4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.133877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f949349-900f-4fed-91a9-924d71ef8b4d admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.134577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f949349-900f-4fed-91a9-924d71ef8b4d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.137066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f949349-900f-4fed-91a9-924d71ef8b4d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.160142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.161040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.161329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.161610 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] PATH_INFO: `/v3/projects/5fbaf674d16546caadc46447071a304f` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.162791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5fbaf674d16546caadc46447071a304f)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.164744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.306169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 933/1852] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/projects => generated 375 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:41.308677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f949349-900f-4fed-91a9-924d71ef8b4d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.310393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1f949349-900f-4fed-91a9-924d71ef8b4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.313486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5fbaf674d16546caadc46447071a304f', 'request_id': 'req-0d173a7d-0815-4307-8d12-53fa8e0c451d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.376557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f949349-900f-4fed-91a9-924d71ef8b4d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.381265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d065385f-cbc5-4817-9037-eb7ff453b25f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.381917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d065385f-cbc5-4817-9037-eb7ff453b25f admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.382197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d065385f-cbc5-4817-9037-eb7ff453b25f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.382473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d065385f-cbc5-4817-9037-eb7ff453b25f admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.383191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d065385f-cbc5-4817-9037-eb7ff453b25f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:41.383498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d065385f-cbc5-4817-9037-eb7ff453b25f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.385125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d065385f-cbc5-4817-9037-eb7ff453b25f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.396542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 934/1853] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:41.397988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456701dc-b708-415f-81e0-eb1c0019a43f None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.400754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456701dc-b708-415f-81e0-eb1c0019a43f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.400754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456701dc-b708-415f-81e0-eb1c0019a43f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.412814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5fbaf674d16546caadc46447071a304f', 'request_id': 'req-0d173a7d-0815-4307-8d12-53fa8e0c451d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.420292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5fbaf674d16546caadc46447071a304f', 'request_id': 'req-0d173a7d-0815-4307-8d12-53fa8e0c451d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.424983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-456701dc-b708-415f-81e0-eb1c0019a43f None None] MFA Rules not processed for user `5fd21861e41b4f43b74bacd53757f628`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:41.426183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-456701dc-b708-415f-81e0-eb1c0019a43f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.431163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] The token cache is being invalidate because project 5fbaf674d16546caadc46447071a304f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:41.431458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.432111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0d173a7d-0815-4307-8d12-53fa8e0c451d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:41.434411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 920/1854] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:41 2022] DELETE /identity/v3/projects/5fbaf674d16546caadc46447071a304f => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:41.436037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c199f81b-7fbd-4198-8776-287ccb3cda75 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.436037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c199f81b-7fbd-4198-8776-287ccb3cda75 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.436233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c199f81b-7fbd-4198-8776-287ccb3cda75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.454853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-c199f81b-7fbd-4198-8776-287ccb3cda75 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:41.456062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c199f81b-7fbd-4198-8776-287ccb3cda75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.464390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 921/1855] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:41.465204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c199f81b-7fbd-4198-8776-287ccb3cda75 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.471483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c199f81b-7fbd-4198-8776-287ccb3cda75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.484198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c199f81b-7fbd-4198-8776-287ccb3cda75 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.489172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11465260-280f-4e81-ac5b-7b04798dd326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.489834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11465260-280f-4e81-ac5b-7b04798dd326 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.490116 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11465260-280f-4e81-ac5b-7b04798dd326 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.490395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11465260-280f-4e81-ac5b-7b04798dd326 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.491186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11465260-280f-4e81-ac5b-7b04798dd326 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.493133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11465260-280f-4e81-ac5b-7b04798dd326 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.497124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 935/1856] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:41.498835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456701dc-b708-415f-81e0-eb1c0019a43f None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.501451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-456701dc-b708-415f-81e0-eb1c0019a43f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.515314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456701dc-b708-415f-81e0-eb1c0019a43f None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.520007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e0024d4-8fa8-4b7f-a8e9-67097f75b4de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.520698 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0024d4-8fa8-4b7f-a8e9-67097f75b4de admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.521037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0024d4-8fa8-4b7f-a8e9-67097f75b4de admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.521319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e0024d4-8fa8-4b7f-a8e9-67097f75b4de admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.522043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0024d4-8fa8-4b7f-a8e9-67097f75b4de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.523956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e0024d4-8fa8-4b7f-a8e9-67097f75b4de admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.581419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 922/1857] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/users => generated 294 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:41.581419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49500d99-b088-4b71-a4c1-c903d8805221 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.581419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49500d99-b088-4b71-a4c1-c903d8805221 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.581419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49500d99-b088-4b71-a4c1-c903d8805221 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.590999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 936/1858] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/projects => generated 363 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:41.592382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b16b78-ec7b-451c-9f82-d7252e8d2abd None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.592788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b16b78-ec7b-451c-9f82-d7252e8d2abd None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.593189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b16b78-ec7b-451c-9f82-d7252e8d2abd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.599411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-49500d99-b088-4b71-a4c1-c903d8805221 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:41.600621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-49500d99-b088-4b71-a4c1-c903d8805221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.612910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e0b16b78-ec7b-451c-9f82-d7252e8d2abd None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:41.614386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e0b16b78-ec7b-451c-9f82-d7252e8d2abd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.664628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 923/1859] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:41.665505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49500d99-b088-4b71-a4c1-c903d8805221 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.666823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-49500d99-b088-4b71-a4c1-c903d8805221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.675585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 937/1860] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:41.676602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b16b78-ec7b-451c-9f82-d7252e8d2abd None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.686745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b16b78-ec7b-451c-9f82-d7252e8d2abd None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.692486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dadac5-70b2-4823-b100-c025bcc76b30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.694274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dadac5-70b2-4823-b100-c025bcc76b30 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.694556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dadac5-70b2-4823-b100-c025bcc76b30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.694844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2dadac5-70b2-4823-b100-c025bcc76b30 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.695593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dadac5-70b2-4823-b100-c025bcc76b30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.697304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2dadac5-70b2-4823-b100-c025bcc76b30 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.707963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 938/1861] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:41.709197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dadac5-70b2-4823-b100-c025bcc76b30 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.721068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2dadac5-70b2-4823-b100-c025bcc76b30 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.726820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afa9c41-179d-4a47-9ee9-45d2fe15224e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.727029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49500d99-b088-4b71-a4c1-c903d8805221 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.728209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afa9c41-179d-4a47-9ee9-45d2fe15224e admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.728562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afa9c41-179d-4a47-9ee9-45d2fe15224e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.728969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afa9c41-179d-4a47-9ee9-45d2fe15224e admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.730396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afa9c41-179d-4a47-9ee9-45d2fe15224e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.731781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b414121-0ce4-41b9-8ffb-11570ac6b89e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.732438 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b414121-0ce4-41b9-8ffb-11570ac6b89e service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.732726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b414121-0ce4-41b9-8ffb-11570ac6b89e service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.732859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afa9c41-179d-4a47-9ee9-45d2fe15224e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.733049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b414121-0ce4-41b9-8ffb-11570ac6b89e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.739180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:41.767796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:41.798922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b414121-0ce4-41b9-8ffb-11570ac6b89e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.800378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b414121-0ce4-41b9-8ffb-11570ac6b89e service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.810166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 924/1862] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:41 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:41.811444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4322573-9f7c-4c5f-8d9b-2e316babeeaf None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.811739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4322573-9f7c-4c5f-8d9b-2e316babeeaf None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.812023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4322573-9f7c-4c5f-8d9b-2e316babeeaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.830001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 939/1863] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/users => generated 282 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:41.831138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afa9c41-179d-4a47-9ee9-45d2fe15224e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.834306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a4322573-9f7c-4c5f-8d9b-2e316babeeaf None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:41.836693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a4322573-9f7c-4c5f-8d9b-2e316babeeaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:41.843389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afa9c41-179d-4a47-9ee9-45d2fe15224e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.848547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 925/1864] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:41.849985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2658a488-00df-46f2-bf63-0ab3bd1a1509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.850857 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2658a488-00df-46f2-bf63-0ab3bd1a1509 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.851161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2658a488-00df-46f2-bf63-0ab3bd1a1509 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.851435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2658a488-00df-46f2-bf63-0ab3bd1a1509 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.852260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2658a488-00df-46f2-bf63-0ab3bd1a1509 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:41.852569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2658a488-00df-46f2-bf63-0ab3bd1a1509 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.854352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2658a488-00df-46f2-bf63-0ab3bd1a1509 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.855386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4322573-9f7c-4c5f-8d9b-2e316babeeaf None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.868367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 940/1865] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:41.874202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4322573-9f7c-4c5f-8d9b-2e316babeeaf None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.874446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2658a488-00df-46f2-bf63-0ab3bd1a1509 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.883283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2658a488-00df-46f2-bf63-0ab3bd1a1509 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.889096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019fd46b-3af6-4796-8745-ba4987d0a95e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.897309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019fd46b-3af6-4796-8745-ba4987d0a95e admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.897309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019fd46b-3af6-4796-8745-ba4987d0a95e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.897309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-019fd46b-3af6-4796-8745-ba4987d0a95e admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.897309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019fd46b-3af6-4796-8745-ba4987d0a95e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.897973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57391f9f-4dcb-4d7a-bc5b-2bdc1600494d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.898077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-019fd46b-3af6-4796-8745-ba4987d0a95e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.898159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57391f9f-4dcb-4d7a-bc5b-2bdc1600494d admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.898159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57391f9f-4dcb-4d7a-bc5b-2bdc1600494d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.898159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57391f9f-4dcb-4d7a-bc5b-2bdc1600494d admin admin] PATH_INFO: `/v3/projects/38c6e46ec17c46b4ae05a4edc5099e8e/users/d9a69c2359bc449f8ebca59bc3be4266/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.910287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 941/1866] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:41 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:41.911152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019fd46b-3af6-4796-8745-ba4987d0a95e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.925601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57391f9f-4dcb-4d7a-bc5b-2bdc1600494d admin admin] RBAC: Authorizing `identity:create_grant(project_id=38c6e46ec17c46b4ae05a4edc5099e8e, user_id=d9a69c2359bc449f8ebca59bc3be4266, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.925765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-019fd46b-3af6-4796-8745-ba4987d0a95e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.925852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57391f9f-4dcb-4d7a-bc5b-2bdc1600494d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.927911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669c61ed-6009-4801-9718-1d96419d6064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.928699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669c61ed-6009-4801-9718-1d96419d6064 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.928999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669c61ed-6009-4801-9718-1d96419d6064 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.929284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669c61ed-6009-4801-9718-1d96419d6064 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.930012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669c61ed-6009-4801-9718-1d96419d6064 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:41.930320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669c61ed-6009-4801-9718-1d96419d6064 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.932141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-669c61ed-6009-4801-9718-1d96419d6064 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.944530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 942/1867] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:41.945501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669c61ed-6009-4801-9718-1d96419d6064 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:41.954677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669c61ed-6009-4801-9718-1d96419d6064 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:41.959247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d364a6-48bd-405f-907e-7d685ac657f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:41.959863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d364a6-48bd-405f-907e-7d685ac657f9 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:41.960149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d364a6-48bd-405f-907e-7d685ac657f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:41.960429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51d364a6-48bd-405f-907e-7d685ac657f9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:41.961147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d364a6-48bd-405f-907e-7d685ac657f9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:41.963020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51d364a6-48bd-405f-907e-7d685ac657f9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:41.995611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 926/1868] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:41 2022] PUT /identity/v3/projects/38c6e46ec17c46b4ae05a4edc5099e8e/users/d9a69c2359bc449f8ebca59bc3be4266/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:41.996577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57391f9f-4dcb-4d7a-bc5b-2bdc1600494d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.060035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57391f9f-4dcb-4d7a-bc5b-2bdc1600494d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.064744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e6a81a-8fa6-48a6-8c89-1717d2486d27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.065431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e6a81a-8fa6-48a6-8c89-1717d2486d27 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.065717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e6a81a-8fa6-48a6-8c89-1717d2486d27 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.065997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e6a81a-8fa6-48a6-8c89-1717d2486d27 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.069282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 943/1869] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:41 2022] POST /identity/v3/projects => generated 387 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:42.069431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e6a81a-8fa6-48a6-8c89-1717d2486d27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.069537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d364a6-48bd-405f-907e-7d685ac657f9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.069632 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e6a81a-8fa6-48a6-8c89-1717d2486d27 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.079767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 927/1870] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:41 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:42.080613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e6a81a-8fa6-48a6-8c89-1717d2486d27 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.090738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e6a81a-8fa6-48a6-8c89-1717d2486d27 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.135487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51d364a6-48bd-405f-907e-7d685ac657f9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.141143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243c53c0-ecb1-4b0d-90a1-787f6241324a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.141869 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243c53c0-ecb1-4b0d-90a1-787f6241324a admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.142143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243c53c0-ecb1-4b0d-90a1-787f6241324a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.142419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243c53c0-ecb1-4b0d-90a1-787f6241324a admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.143169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243c53c0-ecb1-4b0d-90a1-787f6241324a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.145198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243c53c0-ecb1-4b0d-90a1-787f6241324a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.147659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5e96f0-c454-4062-a2ec-8fdee2fa52ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.148338 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5e96f0-c454-4062-a2ec-8fdee2fa52ce admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.148646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5e96f0-c454-4062-a2ec-8fdee2fa52ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.148951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5e96f0-c454-4062-a2ec-8fdee2fa52ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.149673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5e96f0-c454-4062-a2ec-8fdee2fa52ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.151287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd5e96f0-c454-4062-a2ec-8fdee2fa52ce admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.162222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 928/1871] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:42 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:42.163178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5e96f0-c454-4062-a2ec-8fdee2fa52ce admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.172896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5e96f0-c454-4062-a2ec-8fdee2fa52ce admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.177732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732d7b30-18aa-4e4f-99d1-58cbcec7bf4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.178300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732d7b30-18aa-4e4f-99d1-58cbcec7bf4b admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.178584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732d7b30-18aa-4e4f-99d1-58cbcec7bf4b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.178859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732d7b30-18aa-4e4f-99d1-58cbcec7bf4b admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.179524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732d7b30-18aa-4e4f-99d1-58cbcec7bf4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.181448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732d7b30-18aa-4e4f-99d1-58cbcec7bf4b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.268484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 944/1872] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:42 2022] POST /identity/v3/projects => generated 387 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:42.268484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243c53c0-ecb1-4b0d-90a1-787f6241324a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.286782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 929/1873] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:42 2022] POST /identity/v3/users => generated 294 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:42.287536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732d7b30-18aa-4e4f-99d1-58cbcec7bf4b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.297700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732d7b30-18aa-4e4f-99d1-58cbcec7bf4b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.338891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243c53c0-ecb1-4b0d-90a1-787f6241324a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.346785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61866b2d-ea39-42fe-9351-3188f9dcbbc0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.347692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61866b2d-ea39-42fe-9351-3188f9dcbbc0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.347972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61866b2d-ea39-42fe-9351-3188f9dcbbc0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.348243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61866b2d-ea39-42fe-9351-3188f9dcbbc0 admin admin] PATH_INFO: `/v3/projects/48c25574ef6c447b9b0aa4dd366fa7af/users/625f54352dcf4ae5a0bc3641b56c14ae/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.362339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719900f-0fd8-4c47-b860-a236b0369ce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.363554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719900f-0fd8-4c47-b860-a236b0369ce5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.363839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719900f-0fd8-4c47-b860-a236b0369ce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.364114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6719900f-0fd8-4c47-b860-a236b0369ce5 admin admin] PATH_INFO: `/v3/projects/38c6e46ec17c46b4ae05a4edc5099e8e/users/d9a69c2359bc449f8ebca59bc3be4266/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.364681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61866b2d-ea39-42fe-9351-3188f9dcbbc0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=48c25574ef6c447b9b0aa4dd366fa7af, user_id=625f54352dcf4ae5a0bc3641b56c14ae, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.366201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6719900f-0fd8-4c47-b860-a236b0369ce5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38c6e46ec17c46b4ae05a4edc5099e8e, user_id=d9a69c2359bc449f8ebca59bc3be4266, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.367665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61866b2d-ea39-42fe-9351-3188f9dcbbc0 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.368863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6719900f-0fd8-4c47-b860-a236b0369ce5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.380502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 930/1874] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:42 2022] PUT /identity/v3/projects/38c6e46ec17c46b4ae05a4edc5099e8e/users/d9a69c2359bc449f8ebca59bc3be4266/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:42.381374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719900f-0fd8-4c47-b860-a236b0369ce5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.438969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6719900f-0fd8-4c47-b860-a236b0369ce5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.441943 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 945/1875] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:42 2022] PUT /identity/v3/projects/48c25574ef6c447b9b0aa4dd366fa7af/users/625f54352dcf4ae5a0bc3641b56c14ae/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:42.442838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61866b2d-ea39-42fe-9351-3188f9dcbbc0 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.501262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec985b62-b0e2-47d3-96ea-887ece3d3071 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.501951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec985b62-b0e2-47d3-96ea-887ece3d3071 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.502239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec985b62-b0e2-47d3-96ea-887ece3d3071 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.502520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec985b62-b0e2-47d3-96ea-887ece3d3071 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.504402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ec985b62-b0e2-47d3-96ea-887ece3d3071 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:42.505624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61866b2d-ea39-42fe-9351-3188f9dcbbc0 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.510707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354ca82e-ca06-4bc2-b355-8a42133a0a4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.511933 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ca82e-ca06-4bc2-b355-8a42133a0a4b admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.512220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ca82e-ca06-4bc2-b355-8a42133a0a4b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.512501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354ca82e-ca06-4bc2-b355-8a42133a0a4b admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.513233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354ca82e-ca06-4bc2-b355-8a42133a0a4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.515250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354ca82e-ca06-4bc2-b355-8a42133a0a4b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.570304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec985b62-b0e2-47d3-96ea-887ece3d3071 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.571843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec985b62-b0e2-47d3-96ea-887ece3d3071 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.583018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 931/1876] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:42 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:42.583749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec985b62-b0e2-47d3-96ea-887ece3d3071 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.595735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec985b62-b0e2-47d3-96ea-887ece3d3071 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.600054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 946/1877] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:42 2022] POST /identity/v3/users => generated 294 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:42.600054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5df80c-b8e9-474a-99f0-6e698d676228 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.600344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5df80c-b8e9-474a-99f0-6e698d676228 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.600650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5df80c-b8e9-474a-99f0-6e698d676228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.607076 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8064bd9-d5c5-4e95-9964-9d46335c9c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.607814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8064bd9-d5c5-4e95-9964-9d46335c9c71 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.608108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8064bd9-d5c5-4e95-9964-9d46335c9c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.608391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8064bd9-d5c5-4e95-9964-9d46335c9c71 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.609217 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8064bd9-d5c5-4e95-9964-9d46335c9c71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.610846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8064bd9-d5c5-4e95-9964-9d46335c9c71 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.626736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 932/1878] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:42 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:42.628064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6169e5be-c464-49e4-9a34-16ad65bbf244 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.628317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6169e5be-c464-49e4-9a34-16ad65bbf244 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.628579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6169e5be-c464-49e4-9a34-16ad65bbf244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.629076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-db5df80c-b8e9-474a-99f0-6e698d676228 None None] MFA Rules not processed for user `d9a69c2359bc449f8ebca59bc3be4266`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:42.630442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-db5df80c-b8e9-474a-99f0-6e698d676228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:42.665723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-6169e5be-c464-49e4-9a34-16ad65bbf244 None None] MFA Rules not processed for user `625f54352dcf4ae5a0bc3641b56c14ae`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:42.670886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6169e5be-c464-49e4-9a34-16ad65bbf244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:42.710657 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 947/1879] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:42 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:42.711707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5df80c-b8e9-474a-99f0-6e698d676228 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.726348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5df80c-b8e9-474a-99f0-6e698d676228 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.733002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612b3a48-8178-47bb-936b-7960412e184d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.734265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b3a48-8178-47bb-936b-7960412e184d admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.735593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b3a48-8178-47bb-936b-7960412e184d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.735888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b3a48-8178-47bb-936b-7960412e184d admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.736659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612b3a48-8178-47bb-936b-7960412e184d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.739186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612b3a48-8178-47bb-936b-7960412e184d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.754188 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 948/1880] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:42 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:42.756139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612b3a48-8178-47bb-936b-7960412e184d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.770699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612b3a48-8178-47bb-936b-7960412e184d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.773671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 933/1881] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:42 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:42.774455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6169e5be-c464-49e4-9a34-16ad65bbf244 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.775943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6169e5be-c464-49e4-9a34-16ad65bbf244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:42.778980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5893283-7c2a-4c43-8789-5e7e6e72d6d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.780782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5893283-7c2a-4c43-8789-5e7e6e72d6d8 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.780782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5893283-7c2a-4c43-8789-5e7e6e72d6d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.780782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5893283-7c2a-4c43-8789-5e7e6e72d6d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.782130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5893283-7c2a-4c43-8789-5e7e6e72d6d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.784397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5893283-7c2a-4c43-8789-5e7e6e72d6d8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.790230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6169e5be-c464-49e4-9a34-16ad65bbf244 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.795403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ba3290-63c2-4fb0-9556-37e1f8a4b859 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.795719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ba3290-63c2-4fb0-9556-37e1f8a4b859 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.795994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ba3290-63c2-4fb0-9556-37e1f8a4b859 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.796283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ba3290-63c2-4fb0-9556-37e1f8a4b859 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.796854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 949/1882] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:42 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:42.797873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5893283-7c2a-4c43-8789-5e7e6e72d6d8 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.801552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a2ba3290-63c2-4fb0-9556-37e1f8a4b859 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:42.808935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5893283-7c2a-4c43-8789-5e7e6e72d6d8 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.814032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ceb91d2-2a82-4c9a-9f49-5069901e39fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.814792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceb91d2-2a82-4c9a-9f49-5069901e39fd admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.815169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceb91d2-2a82-4c9a-9f49-5069901e39fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.815522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceb91d2-2a82-4c9a-9f49-5069901e39fd admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.816431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ceb91d2-2a82-4c9a-9f49-5069901e39fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ceb91d2-2a82-4c9a-9f49-5069901e39fd admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.829654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 950/1883] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:42 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:42.830498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ceb91d2-2a82-4c9a-9f49-5069901e39fd admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.840124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ceb91d2-2a82-4c9a-9f49-5069901e39fd admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.844600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f534a1a3-c83b-43f7-b6ce-c4f66a581884 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.845451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f534a1a3-c83b-43f7-b6ce-c4f66a581884 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.845739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f534a1a3-c83b-43f7-b6ce-c4f66a581884 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.846014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f534a1a3-c83b-43f7-b6ce-c4f66a581884 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.846687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f534a1a3-c83b-43f7-b6ce-c4f66a581884 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.848568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f534a1a3-c83b-43f7-b6ce-c4f66a581884 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.865082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ba3290-63c2-4fb0-9556-37e1f8a4b859 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.866561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ba3290-63c2-4fb0-9556-37e1f8a4b859 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.876484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 934/1884] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:42 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:42.878448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ba3290-63c2-4fb0-9556-37e1f8a4b859 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:42.889025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ba3290-63c2-4fb0-9556-37e1f8a4b859 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:42.893884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059732b9-1ab1-4361-91fb-5d90df96d747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:42.894767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059732b9-1ab1-4361-91fb-5d90df96d747 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:42.895541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059732b9-1ab1-4361-91fb-5d90df96d747 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:42.895541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059732b9-1ab1-4361-91fb-5d90df96d747 admin admin] PATH_INFO: `/v3/projects/15be07d4e645464d9febe68cb3ce29fd/users/b1b320d6bbe9464b9cf9883c245f6354/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:42.910230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059732b9-1ab1-4361-91fb-5d90df96d747 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15be07d4e645464d9febe68cb3ce29fd, user_id=b1b320d6bbe9464b9cf9883c245f6354, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:42.913136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-059732b9-1ab1-4361-91fb-5d90df96d747 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:42.963642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 951/1885] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:42 2022] POST /identity/v3/projects => generated 365 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:42.974126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f534a1a3-c83b-43f7-b6ce-c4f66a581884 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.019498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 935/1886] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:42 2022] PUT /identity/v3/projects/15be07d4e645464d9febe68cb3ce29fd/users/b1b320d6bbe9464b9cf9883c245f6354/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:43.019498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059732b9-1ab1-4361-91fb-5d90df96d747 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.049073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f534a1a3-c83b-43f7-b6ce-c4f66a581884 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.091244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059732b9-1ab1-4361-91fb-5d90df96d747 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.101744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c0ab03-c11f-4b60-a6ed-860d7408c289 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.102488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c0ab03-c11f-4b60-a6ed-860d7408c289 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.102841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c0ab03-c11f-4b60-a6ed-860d7408c289 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.103348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c0ab03-c11f-4b60-a6ed-860d7408c289 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.104104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c0ab03-c11f-4b60-a6ed-860d7408c289 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.111757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70c0ab03-c11f-4b60-a6ed-860d7408c289 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.131049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5297382-ef08-45d1-b0c0-c410a00fff6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.131941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5297382-ef08-45d1-b0c0-c410a00fff6a admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.132222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5297382-ef08-45d1-b0c0-c410a00fff6a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.132523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5297382-ef08-45d1-b0c0-c410a00fff6a admin admin] PATH_INFO: `/v3/projects/c36a3d54680846eea8ab292610a1d8c7/users/4f3722cfdfdd4164a3c3ebedb2ac9803/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.146499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5297382-ef08-45d1-b0c0-c410a00fff6a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c36a3d54680846eea8ab292610a1d8c7, user_id=4f3722cfdfdd4164a3c3ebedb2ac9803, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.149211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5297382-ef08-45d1-b0c0-c410a00fff6a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.219415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 936/1887] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:43 2022] POST /identity/v3/users => generated 289 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:43.222998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017a6b51-6fdb-43e3-ac99-f99c01b5f2f8 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.222998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017a6b51-6fdb-43e3-ac99-f99c01b5f2f8 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.222998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017a6b51-6fdb-43e3-ac99-f99c01b5f2f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.246875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 952/1888] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:42 2022] PUT /identity/v3/projects/c36a3d54680846eea8ab292610a1d8c7/users/4f3722cfdfdd4164a3c3ebedb2ac9803/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:43.247781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5297382-ef08-45d1-b0c0-c410a00fff6a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.251084 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-017a6b51-6fdb-43e3-ac99-f99c01b5f2f8 None None] MFA Rules not processed for user `b1b320d6bbe9464b9cf9883c245f6354`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:43.253203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-017a6b51-6fdb-43e3-ac99-f99c01b5f2f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:43.268033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:43.310468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:43.327555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5297382-ef08-45d1-b0c0-c410a00fff6a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.335141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036739dd-87ec-420f-bff4-9f0f0b8329d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.335950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036739dd-87ec-420f-bff4-9f0f0b8329d2 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.341412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 937/1889] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:43 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:43.342334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036739dd-87ec-420f-bff4-9f0f0b8329d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.342334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036739dd-87ec-420f-bff4-9f0f0b8329d2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.342334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036739dd-87ec-420f-bff4-9f0f0b8329d2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.342704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9adea98-dd6b-485a-877f-aea78cd89eba None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.342704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9adea98-dd6b-485a-877f-aea78cd89eba None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.342704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9adea98-dd6b-485a-877f-aea78cd89eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.343111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036739dd-87ec-420f-bff4-9f0f0b8329d2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.353131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 953/1890] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:43 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:43.354324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036739dd-87ec-420f-bff4-9f0f0b8329d2 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.365077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036739dd-87ec-420f-bff4-9f0f0b8329d2 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.367245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-f9adea98-dd6b-485a-877f-aea78cd89eba None None] MFA Rules not processed for user `4f3722cfdfdd4164a3c3ebedb2ac9803`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:43.368533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f9adea98-dd6b-485a-877f-aea78cd89eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:43.369929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de7f6a6-2a0b-4214-b428-0cad922f4513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.370636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de7f6a6-2a0b-4214-b428-0cad922f4513 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.373121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de7f6a6-2a0b-4214-b428-0cad922f4513 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.373121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de7f6a6-2a0b-4214-b428-0cad922f4513 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.373121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de7f6a6-2a0b-4214-b428-0cad922f4513 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.373803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de7f6a6-2a0b-4214-b428-0cad922f4513 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.451553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 938/1891] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:43 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:43.451553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9adea98-dd6b-485a-877f-aea78cd89eba None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.462071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9adea98-dd6b-485a-877f-aea78cd89eba None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.468029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53115c3d-cb35-4162-a738-beae1ce32c4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.469322 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53115c3d-cb35-4162-a738-beae1ce32c4b admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.469599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53115c3d-cb35-4162-a738-beae1ce32c4b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.469949 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53115c3d-cb35-4162-a738-beae1ce32c4b admin admin] PATH_INFO: `/v3/projects/71fbd80ef4fa46fdb8dccec1d7889132/users/6d610ef444c14efaa43dbd2ed0b667f6/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.474819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 954/1892] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:43 2022] POST /identity/v3/projects => generated 387 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:43.476553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de7f6a6-2a0b-4214-b428-0cad922f4513 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.494401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53115c3d-cb35-4162-a738-beae1ce32c4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=71fbd80ef4fa46fdb8dccec1d7889132, user_id=6d610ef444c14efaa43dbd2ed0b667f6, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.498117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53115c3d-cb35-4162-a738-beae1ce32c4b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.616442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de7f6a6-2a0b-4214-b428-0cad922f4513 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.641730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 939/1893] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:43 2022] PUT /identity/v3/projects/71fbd80ef4fa46fdb8dccec1d7889132/users/6d610ef444c14efaa43dbd2ed0b667f6/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:43.641840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b28288-10d5-42d6-a455-99e921fc1c9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.641939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53115c3d-cb35-4162-a738-beae1ce32c4b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.642010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b28288-10d5-42d6-a455-99e921fc1c9a admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.642010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b28288-10d5-42d6-a455-99e921fc1c9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.642010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b28288-10d5-42d6-a455-99e921fc1c9a admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.642010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b28288-10d5-42d6-a455-99e921fc1c9a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.642010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b28288-10d5-42d6-a455-99e921fc1c9a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.714992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53115c3d-cb35-4162-a738-beae1ce32c4b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.721913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fed5da-cb10-4b00-90ce-bdeb2f22b4b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.722747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fed5da-cb10-4b00-90ce-bdeb2f22b4b8 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.723475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fed5da-cb10-4b00-90ce-bdeb2f22b4b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.723979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fed5da-cb10-4b00-90ce-bdeb2f22b4b8 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.725983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fed5da-cb10-4b00-90ce-bdeb2f22b4b8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.730936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fed5da-cb10-4b00-90ce-bdeb2f22b4b8 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.765134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 955/1894] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:43 2022] POST /identity/v3/projects => generated 387 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:43.765134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b28288-10d5-42d6-a455-99e921fc1c9a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.818351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 940/1895] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:43 2022] POST /identity/v3/users => generated 300 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:43.819557 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fed5da-cb10-4b00-90ce-bdeb2f22b4b8 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.829786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fed5da-cb10-4b00-90ce-bdeb2f22b4b8 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.830927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b28288-10d5-42d6-a455-99e921fc1c9a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.834599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0790625f-95ef-413a-96b0-c242d8c0b6c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.835330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0790625f-95ef-413a-96b0-c242d8c0b6c3 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.835609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0790625f-95ef-413a-96b0-c242d8c0b6c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.835909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0790625f-95ef-413a-96b0-c242d8c0b6c3 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.836592 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0790625f-95ef-413a-96b0-c242d8c0b6c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.837024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc993d32-3d9b-4033-aa28-f2fa4526960d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.837814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc993d32-3d9b-4033-aa28-f2fa4526960d admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.838228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc993d32-3d9b-4033-aa28-f2fa4526960d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.838611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc993d32-3d9b-4033-aa28-f2fa4526960d admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.840378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0790625f-95ef-413a-96b0-c242d8c0b6c3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.840752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc993d32-3d9b-4033-aa28-f2fa4526960d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.842584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc993d32-3d9b-4033-aa28-f2fa4526960d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.853013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 956/1896] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:43 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:43.853837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc993d32-3d9b-4033-aa28-f2fa4526960d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.865829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc993d32-3d9b-4033-aa28-f2fa4526960d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.871107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad90beaa-904b-45cc-96a5-41c66a5ebaed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.874561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad90beaa-904b-45cc-96a5-41c66a5ebaed admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.874561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad90beaa-904b-45cc-96a5-41c66a5ebaed admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.874561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad90beaa-904b-45cc-96a5-41c66a5ebaed admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.874561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad90beaa-904b-45cc-96a5-41c66a5ebaed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.874855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad90beaa-904b-45cc-96a5-41c66a5ebaed admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.884131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 957/1897] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:43.884905 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad90beaa-904b-45cc-96a5-41c66a5ebaed admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.894616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad90beaa-904b-45cc-96a5-41c66a5ebaed admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.899302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd05b1-7088-479e-9b61-1dee4166bbbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.900155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd05b1-7088-479e-9b61-1dee4166bbbc admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.900440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd05b1-7088-479e-9b61-1dee4166bbbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.900719 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd05b1-7088-479e-9b61-1dee4166bbbc admin admin] PATH_INFO: `/v3/projects/71fbd80ef4fa46fdb8dccec1d7889132/users/6d610ef444c14efaa43dbd2ed0b667f6/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.902799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cd05b1-7088-479e-9b61-1dee4166bbbc admin admin] RBAC: Authorizing `identity:create_grant(project_id=71fbd80ef4fa46fdb8dccec1d7889132, user_id=6d610ef444c14efaa43dbd2ed0b667f6, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.905530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cd05b1-7088-479e-9b61-1dee4166bbbc admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:43.915606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 941/1898] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:43 2022] POST /identity/v3/users => generated 300 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:43.916554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 958/1899] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:43 2022] PUT /identity/v3/projects/71fbd80ef4fa46fdb8dccec1d7889132/users/6d610ef444c14efaa43dbd2ed0b667f6/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:43.919713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0790625f-95ef-413a-96b0-c242d8c0b6c3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.925795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd05b1-7088-479e-9b61-1dee4166bbbc admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:43.929580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0790625f-95ef-413a-96b0-c242d8c0b6c3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.987618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd05b1-7088-479e-9b61-1dee4166bbbc admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:43.992545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ddf2698-a2ba-4c8a-b4f5-f4b094497aa4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:43.993262 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddf2698-a2ba-4c8a-b4f5-f4b094497aa4 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:43.993541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddf2698-a2ba-4c8a-b4f5-f4b094497aa4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:43.993821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ddf2698-a2ba-4c8a-b4f5-f4b094497aa4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:43.994554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ddf2698-a2ba-4c8a-b4f5-f4b094497aa4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:43.996699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ddf2698-a2ba-4c8a-b4f5-f4b094497aa4 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.001536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da30f2-479f-4f57-b6b7-419b83d095d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.002454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da30f2-479f-4f57-b6b7-419b83d095d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.003151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da30f2-479f-4f57-b6b7-419b83d095d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.003487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53da30f2-479f-4f57-b6b7-419b83d095d2 admin admin] PATH_INFO: `/v3/projects/f98c2458f1ef45859daaeee855e5ffcf/users/c4e957fd0af74489affc53129529cb9e/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.007007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 959/1900] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:43 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:44.008264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac51406-a906-4c2f-bbff-4eef3d29b0f4 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.008557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac51406-a906-4c2f-bbff-4eef3d29b0f4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.008862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac51406-a906-4c2f-bbff-4eef3d29b0f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.019444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da30f2-479f-4f57-b6b7-419b83d095d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f98c2458f1ef45859daaeee855e5ffcf, user_id=c4e957fd0af74489affc53129529cb9e, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.022378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53da30f2-479f-4f57-b6b7-419b83d095d2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.034888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-2ac51406-a906-4c2f-bbff-4eef3d29b0f4 None None] MFA Rules not processed for user `6d610ef444c14efaa43dbd2ed0b667f6`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:44.036125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2ac51406-a906-4c2f-bbff-4eef3d29b0f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:44.101055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 960/1901] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:44 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:44.101949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac51406-a906-4c2f-bbff-4eef3d29b0f4 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:44.111436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac51406-a906-4c2f-bbff-4eef3d29b0f4 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:44.116471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c666da-7d68-474a-9d78-bc3d068681a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.117386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c666da-7d68-474a-9d78-bc3d068681a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.117668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c666da-7d68-474a-9d78-bc3d068681a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.117950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08c666da-7d68-474a-9d78-bc3d068681a4 admin admin] PATH_INFO: `/v3/projects/3d39d69fbaa24f6e8b6e6af8f2198733/users/c2b9dab177c84b53b51a9999f4adf543/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.132966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 942/1902] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:43 2022] PUT /identity/v3/projects/f98c2458f1ef45859daaeee855e5ffcf/users/c4e957fd0af74489affc53129529cb9e/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:44.133679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da30f2-479f-4f57-b6b7-419b83d095d2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:44.137596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c666da-7d68-474a-9d78-bc3d068681a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d39d69fbaa24f6e8b6e6af8f2198733, user_id=c2b9dab177c84b53b51a9999f4adf543, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.141000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08c666da-7d68-474a-9d78-bc3d068681a4 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.191242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53da30f2-479f-4f57-b6b7-419b83d095d2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:44.194912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17304a0-6a27-4b00-9dcb-fdcbacd4840d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.195557 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17304a0-6a27-4b00-9dcb-fdcbacd4840d service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.195845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17304a0-6a27-4b00-9dcb-fdcbacd4840d service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.196472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17304a0-6a27-4b00-9dcb-fdcbacd4840d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.265986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17304a0-6a27-4b00-9dcb-fdcbacd4840d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.267560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17304a0-6a27-4b00-9dcb-fdcbacd4840d service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.278345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 943/1903] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:44 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:44.279151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17304a0-6a27-4b00-9dcb-fdcbacd4840d service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:44.340526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17304a0-6a27-4b00-9dcb-fdcbacd4840d service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:44.346273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da8419d-02c7-4e38-8f82-5129513f13cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.346943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da8419d-02c7-4e38-8f82-5129513f13cd admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.347237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da8419d-02c7-4e38-8f82-5129513f13cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.347511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da8419d-02c7-4e38-8f82-5129513f13cd admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.348238 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da8419d-02c7-4e38-8f82-5129513f13cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.349946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da8419d-02c7-4e38-8f82-5129513f13cd admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.350087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 961/1904] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:44 2022] PUT /identity/v3/projects/3d39d69fbaa24f6e8b6e6af8f2198733/users/c2b9dab177c84b53b51a9999f4adf543/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:44.354176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c666da-7d68-474a-9d78-bc3d068681a4 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:44.363920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 944/1905] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:44 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:44.364946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da8419d-02c7-4e38-8f82-5129513f13cd admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:44.375246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da8419d-02c7-4e38-8f82-5129513f13cd admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:44.415768 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08c666da-7d68-474a-9d78-bc3d068681a4 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:44.425048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc376d6f-5179-4c92-96f4-5f8733fc0513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.425898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc376d6f-5179-4c92-96f4-5f8733fc0513 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.426617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc376d6f-5179-4c92-96f4-5f8733fc0513 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.426617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc376d6f-5179-4c92-96f4-5f8733fc0513 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.427508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc376d6f-5179-4c92-96f4-5f8733fc0513 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.429442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc376d6f-5179-4c92-96f4-5f8733fc0513 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.433436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633ceeea-7eac-44cd-bf5e-5f6c4b193bfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.437656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633ceeea-7eac-44cd-bf5e-5f6c4b193bfe admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.437656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633ceeea-7eac-44cd-bf5e-5f6c4b193bfe admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.437656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633ceeea-7eac-44cd-bf5e-5f6c4b193bfe admin admin] PATH_INFO: `/v3/projects/f98c2458f1ef45859daaeee855e5ffcf/users/c4e957fd0af74489affc53129529cb9e/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.437656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633ceeea-7eac-44cd-bf5e-5f6c4b193bfe admin admin] RBAC: Authorizing `identity:create_grant(project_id=f98c2458f1ef45859daaeee855e5ffcf, user_id=c4e957fd0af74489affc53129529cb9e, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.439631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633ceeea-7eac-44cd-bf5e-5f6c4b193bfe admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.442187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 962/1906] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:44 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:05:44.446106 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc376d6f-5179-4c92-96f4-5f8733fc0513 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:44.454027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 945/1907] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:44 2022] PUT /identity/v3/projects/f98c2458f1ef45859daaeee855e5ffcf/users/c4e957fd0af74489affc53129529cb9e/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:44.456835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc376d6f-5179-4c92-96f4-5f8733fc0513 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:44.464150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc29cd2d-51fb-4885-9e6d-8c467acec64d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.464474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc29cd2d-51fb-4885-9e6d-8c467acec64d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.464758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc29cd2d-51fb-4885-9e6d-8c467acec64d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.491230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bc29cd2d-51fb-4885-9e6d-8c467acec64d None None] MFA Rules not processed for user `c4e957fd0af74489affc53129529cb9e`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:44.492448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bc29cd2d-51fb-4885-9e6d-8c467acec64d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:44.515196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d28ddc-0342-4e5d-947c-830e84ce3171 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.516187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d28ddc-0342-4e5d-947c-830e84ce3171 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.516382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d28ddc-0342-4e5d-947c-830e84ce3171 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.516816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d28ddc-0342-4e5d-947c-830e84ce3171 admin admin] PATH_INFO: `/v3/projects/3d39d69fbaa24f6e8b6e6af8f2198733/users/c2b9dab177c84b53b51a9999f4adf543/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.518911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d28ddc-0342-4e5d-947c-830e84ce3171 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3d39d69fbaa24f6e8b6e6af8f2198733, user_id=c2b9dab177c84b53b51a9999f4adf543, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.521907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d28ddc-0342-4e5d-947c-830e84ce3171 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.533521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 963/1908] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:44 2022] PUT /identity/v3/projects/3d39d69fbaa24f6e8b6e6af8f2198733/users/c2b9dab177c84b53b51a9999f4adf543/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:44.548541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135a82a-0bac-4eae-82e9-99d5f61de3b9 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.552746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135a82a-0bac-4eae-82e9-99d5f61de3b9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.553154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135a82a-0bac-4eae-82e9-99d5f61de3b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.579967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 946/1909] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:44 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:44.580870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc29cd2d-51fb-4885-9e6d-8c467acec64d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:44.589459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-1135a82a-0bac-4eae-82e9-99d5f61de3b9 None None] MFA Rules not processed for user `c2b9dab177c84b53b51a9999f4adf543`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:44.590824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1135a82a-0bac-4eae-82e9-99d5f61de3b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:44.646951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc29cd2d-51fb-4885-9e6d-8c467acec64d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:44.657998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:44.682611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 964/1910] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:44 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:44.683377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1135a82a-0bac-4eae-82e9-99d5f61de3b9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:44.699067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9a5909-ff61-446a-be24-94e4cffdbd70 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.700205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9a5909-ff61-446a-be24-94e4cffdbd70 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.700205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9a5909-ff61-446a-be24-94e4cffdbd70 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.700409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9a5909-ff61-446a-be24-94e4cffdbd70 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.739456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1135a82a-0bac-4eae-82e9-99d5f61de3b9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:44.743295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091db36e-131e-424c-ba5c-ba989a1b2260 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.743948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091db36e-131e-424c-ba5c-ba989a1b2260 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.744236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091db36e-131e-424c-ba5c-ba989a1b2260 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.744520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091db36e-131e-424c-ba5c-ba989a1b2260 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.761878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9a5909-ff61-446a-be24-94e4cffdbd70 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.764120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9a5909-ff61-446a-be24-94e4cffdbd70 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.777739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 947/1911] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:44 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:44.817986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091db36e-131e-424c-ba5c-ba989a1b2260 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.819562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091db36e-131e-424c-ba5c-ba989a1b2260 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:44.837146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 965/1912] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:44 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:44.860522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9a5909-ff61-446a-be24-94e4cffdbd70 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:44.948981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9a5909-ff61-446a-be24-94e4cffdbd70 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:44.954380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:44.955288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:44.962291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:44.962291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] PATH_INFO: `/v3/users/b1b320d6bbe9464b9cf9883c245f6354` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:44.962291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1b320d6bbe9464b9cf9883c245f6354)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:44.962291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:45.310222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1b320d6bbe9464b9cf9883c245f6354'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:45.331999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1b320d6bbe9464b9cf9883c245f6354'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:45.331999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1b320d6bbe9464b9cf9883c245f6354'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:45.336649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1b320d6bbe9464b9cf9883c245f6354'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:45.358299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091db36e-131e-424c-ba5c-ba989a1b2260 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:45.368068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091db36e-131e-424c-ba5c-ba989a1b2260 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:45.373044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b8cc73-d308-47ce-a930-182e273ce47d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:45.373690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b8cc73-d308-47ce-a930-182e273ce47d service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:45.373972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b8cc73-d308-47ce-a930-182e273ce47d service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:45.374248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b8cc73-d308-47ce-a930-182e273ce47d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:45.376054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c3b8cc73-d308-47ce-a930-182e273ce47d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:45.388847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b8cc73-d308-47ce-a930-182e273ce47d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:45.390378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b8cc73-d308-47ce-a930-182e273ce47d service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:45.400580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 966/1913] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:45 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:45.469445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1b320d6bbe9464b9cf9883c245f6354'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:45.493221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 948/1914] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:44 2022] DELETE /identity/v3/users/b1b320d6bbe9464b9cf9883c245f6354 => generated 0 bytes in 633 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:45.498502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f890b748-a49d-4ceb-8b77-04f7b90a0bea None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:45.498948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f890b748-a49d-4ceb-8b77-04f7b90a0bea None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:45.499373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f890b748-a49d-4ceb-8b77-04f7b90a0bea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:45.524942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-f890b748-a49d-4ceb-8b77-04f7b90a0bea None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:45.526925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f890b748-a49d-4ceb-8b77-04f7b90a0bea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:45.593543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 967/1915] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:45.600811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:45.602219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:45.665485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e37c69b3-9742-42fc-bab9-3fd3adfd2480 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:45.669870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84dfab7-5268-4f23-a8e6-90707007f901 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:45.670518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84dfab7-5268-4f23-a8e6-90707007f901 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:45.670805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84dfab7-5268-4f23-a8e6-90707007f901 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:45.671085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84dfab7-5268-4f23-a8e6-90707007f901 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:45.680540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84dfab7-5268-4f23-a8e6-90707007f901 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:45.682028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:45.683709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84dfab7-5268-4f23-a8e6-90707007f901 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:45.693478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 949/1916] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:46.317607 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f890b748-a49d-4ceb-8b77-04f7b90a0bea None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:46.318936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f890b748-a49d-4ceb-8b77-04f7b90a0bea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:46.335694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f890b748-a49d-4ceb-8b77-04f7b90a0bea None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:46.341248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:46.345529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:46.345942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:46.346328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] PATH_INFO: `/v3/projects/15be07d4e645464d9febe68cb3ce29fd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:46.347760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] RBAC: Authorizing `identity:delete_project(project_id=15be07d4e645464d9febe68cb3ce29fd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:46.349952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:46.472177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15be07d4e645464d9febe68cb3ce29fd', 'request_id': 'req-d32228f4-a380-4ce0-a484-4c5b81b80e5d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:46.475986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15be07d4e645464d9febe68cb3ce29fd', 'request_id': 'req-d32228f4-a380-4ce0-a484-4c5b81b80e5d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:46.482147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15be07d4e645464d9febe68cb3ce29fd', 'request_id': 'req-d32228f4-a380-4ce0-a484-4c5b81b80e5d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:46.645886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] The token cache is being invalidate because project 15be07d4e645464d9febe68cb3ce29fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:46.646297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:46.647286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:46.649651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 968/1917] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:46 2022] DELETE /identity/v3/projects/15be07d4e645464d9febe68cb3ce29fd => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:46.653418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84dfab7-5268-4f23-a8e6-90707007f901 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:46.665956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84dfab7-5268-4f23-a8e6-90707007f901 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:46.735762 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:46.736319 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:46.736573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:46.736904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] PATH_INFO: `/v3/users/c4e957fd0af74489affc53129529cb9e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:46.738518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4e957fd0af74489affc53129529cb9e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:46.740350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:46.987441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4e957fd0af74489affc53129529cb9e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:46.988339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4e957fd0af74489affc53129529cb9e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:46.988968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4e957fd0af74489affc53129529cb9e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:46.995778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4e957fd0af74489affc53129529cb9e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:47.107824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:47.109446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:47.122098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32228f4-a380-4ce0-a484-4c5b81b80e5d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:47.126578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:47.127433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:47.127714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:47.127988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] PATH_INFO: `/v3/users/4f3722cfdfdd4164a3c3ebedb2ac9803` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:47.129193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] RBAC: Authorizing `identity:delete_user(user_id=4f3722cfdfdd4164a3c3ebedb2ac9803)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:47.131156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:47.201186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-abcdf7bc-197d-4cbf-bbac-ce2f1f57444b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4e957fd0af74489affc53129529cb9e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:47.220223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 950/1918] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:46 2022] DELETE /identity/v3/users/c4e957fd0af74489affc53129529cb9e => generated 0 bytes in 568 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:47.544346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4f3722cfdfdd4164a3c3ebedb2ac9803'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:47.544346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4f3722cfdfdd4164a3c3ebedb2ac9803'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:47.549250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4f3722cfdfdd4164a3c3ebedb2ac9803'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:47.726072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4f3722cfdfdd4164a3c3ebedb2ac9803'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:47.726712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4f3722cfdfdd4164a3c3ebedb2ac9803'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:47.756320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 969/1919] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:47 2022] DELETE /identity/v3/users/4f3722cfdfdd4164a3c3ebedb2ac9803 => generated 0 bytes in 649 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:47.761074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b524d8-a6b2-49b1-b85d-4071e6a2b769 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:47.761559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b524d8-a6b2-49b1-b85d-4071e6a2b769 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:47.762494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b524d8-a6b2-49b1-b85d-4071e6a2b769 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:47.785613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-69b524d8-a6b2-49b1-b85d-4071e6a2b769 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:47.786930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-69b524d8-a6b2-49b1-b85d-4071e6a2b769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:47.862061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 951/1920] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:47.872247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:47.873746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:47.899373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b524d8-a6b2-49b1-b85d-4071e6a2b769 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:47.904118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-69b524d8-a6b2-49b1-b85d-4071e6a2b769 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:47.921078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b524d8-a6b2-49b1-b85d-4071e6a2b769 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:47.925849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:47.926685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:47.926968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:47.927274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] PATH_INFO: `/v3/projects/f98c2458f1ef45859daaeee855e5ffcf` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:47.928498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f98c2458f1ef45859daaeee855e5ffcf)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:47.930511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:47.938628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629b708d-5329-4ed7-9b50-cf61a590bdff admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:47.942160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3515c6b3-2240-44b0-93bb-abc9e4243535 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:47.942808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3515c6b3-2240-44b0-93bb-abc9e4243535 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:47.943094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3515c6b3-2240-44b0-93bb-abc9e4243535 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:47.943374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3515c6b3-2240-44b0-93bb-abc9e4243535 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:47.952942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3515c6b3-2240-44b0-93bb-abc9e4243535 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:47.954460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3515c6b3-2240-44b0-93bb-abc9e4243535 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:47.963868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 970/1921] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:47.972584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:48.019623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:48.037821 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f98c2458f1ef45859daaeee855e5ffcf', 'request_id': 'req-f2e897dc-77ae-42a6-9687-57de1935d98f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.137520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f98c2458f1ef45859daaeee855e5ffcf', 'request_id': 'req-f2e897dc-77ae-42a6-9687-57de1935d98f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.144270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f98c2458f1ef45859daaeee855e5ffcf', 'request_id': 'req-f2e897dc-77ae-42a6-9687-57de1935d98f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.154626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] The token cache is being invalidate because project f98c2458f1ef45859daaeee855e5ffcf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:48.154937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.155602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f2e897dc-77ae-42a6-9687-57de1935d98f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.157210 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 952/1922] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:47 2022] DELETE /identity/v3/projects/f98c2458f1ef45859daaeee855e5ffcf => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:48.162360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ccc60f-2c5c-44ae-923f-a570c962503d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.162728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ccc60f-2c5c-44ae-923f-a570c962503d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.163018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ccc60f-2c5c-44ae-923f-a570c962503d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.185325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c1ccc60f-2c5c-44ae-923f-a570c962503d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:48.186579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c1ccc60f-2c5c-44ae-923f-a570c962503d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:48.248189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 971/1923] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:48.255777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907940ac-b48d-49b4-926a-6b6a983efb09 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.256021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907940ac-b48d-49b4-926a-6b6a983efb09 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.256151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907940ac-b48d-49b4-926a-6b6a983efb09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.274522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-907940ac-b48d-49b4-926a-6b6a983efb09 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:48.275741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-907940ac-b48d-49b4-926a-6b6a983efb09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:48.283968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 953/1924] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:48.289985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ccc60f-2c5c-44ae-923f-a570c962503d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:48.303341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ccc60f-2c5c-44ae-923f-a570c962503d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:48.308048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:48.309298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.309586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.309951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.310579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:48.310889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:48.312512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:48.325090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 972/1925] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:48.329354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907940ac-b48d-49b4-926a-6b6a983efb09 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:48.338748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907940ac-b48d-49b4-926a-6b6a983efb09 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:48.343992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfae12f3-c500-45f4-affd-2687e1fc8cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:48.344642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfae12f3-c500-45f4-affd-2687e1fc8cd4 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.344958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfae12f3-c500-45f4-affd-2687e1fc8cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.345234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfae12f3-c500-45f4-affd-2687e1fc8cd4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.345926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfae12f3-c500-45f4-affd-2687e1fc8cd4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:48.347816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfae12f3-c500-45f4-affd-2687e1fc8cd4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:48.437298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:48.439108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:48.448631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 954/1926] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:48 2022] POST /identity/v3/projects => generated 379 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:05:48.451878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfae12f3-c500-45f4-affd-2687e1fc8cd4 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:48.453720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d63b4c-bf36-4bf0-8ef4-5ecc159a176e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:48.517223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:48.518486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.518834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.519057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] PATH_INFO: `/v3/projects/c36a3d54680846eea8ab292610a1d8c7` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.520446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c36a3d54680846eea8ab292610a1d8c7)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:48.522470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:48.526096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfae12f3-c500-45f4-affd-2687e1fc8cd4 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:48.530993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53a3f18-6634-403b-906a-f65d9e47ec87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:48.531756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53a3f18-6634-403b-906a-f65d9e47ec87 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.532016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53a3f18-6634-403b-906a-f65d9e47ec87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.532274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f53a3f18-6634-403b-906a-f65d9e47ec87 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.532994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53a3f18-6634-403b-906a-f65d9e47ec87 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:48.534997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f53a3f18-6634-403b-906a-f65d9e47ec87 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:48.628915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 955/1927] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:48 2022] POST /identity/v3/users => generated 290 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:48.628915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53a3f18-6634-403b-906a-f65d9e47ec87 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:48.638927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f53a3f18-6634-403b-906a-f65d9e47ec87 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:48.643923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2775cac-8d6c-4c70-aa45-940082d4adb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:48.644886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2775cac-8d6c-4c70-aa45-940082d4adb8 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.650060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2775cac-8d6c-4c70-aa45-940082d4adb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.650060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2775cac-8d6c-4c70-aa45-940082d4adb8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.650060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2775cac-8d6c-4c70-aa45-940082d4adb8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:48.650060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2775cac-8d6c-4c70-aa45-940082d4adb8 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:48.664409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 956/1928] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:48 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:05:48.665469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2775cac-8d6c-4c70-aa45-940082d4adb8 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:48.676013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2775cac-8d6c-4c70-aa45-940082d4adb8 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:48.680848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca60d268-55a2-49ce-ac43-8ba4b3102561 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:48.681712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca60d268-55a2-49ce-ac43-8ba4b3102561 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.682003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca60d268-55a2-49ce-ac43-8ba4b3102561 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.682336 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca60d268-55a2-49ce-ac43-8ba4b3102561 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.683074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca60d268-55a2-49ce-ac43-8ba4b3102561 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:48.685350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca60d268-55a2-49ce-ac43-8ba4b3102561 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:48.693827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c36a3d54680846eea8ab292610a1d8c7', 'request_id': 'req-0ef73eee-baba-4a20-82c4-4e341b6fbac9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.696013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 957/1929] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:48 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:05:48.700671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca60d268-55a2-49ce-ac43-8ba4b3102561 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:48.704066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c36a3d54680846eea8ab292610a1d8c7', 'request_id': 'req-0ef73eee-baba-4a20-82c4-4e341b6fbac9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.709713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c36a3d54680846eea8ab292610a1d8c7', 'request_id': 'req-0ef73eee-baba-4a20-82c4-4e341b6fbac9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.710310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca60d268-55a2-49ce-ac43-8ba4b3102561 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:48.765766 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854eff38-4825-42e2-83f5-718ca66aa8bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:48.766644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854eff38-4825-42e2-83f5-718ca66aa8bd admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.766921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854eff38-4825-42e2-83f5-718ca66aa8bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.767199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854eff38-4825-42e2-83f5-718ca66aa8bd admin admin] PATH_INFO: `/v3/projects/0d771f49f6b641998dc3bc54229534bb/users/d7a57249572345429ba6e706b22a7c5d/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.781043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854eff38-4825-42e2-83f5-718ca66aa8bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d771f49f6b641998dc3bc54229534bb, user_id=d7a57249572345429ba6e706b22a7c5d, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:48.783721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854eff38-4825-42e2-83f5-718ca66aa8bd admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:48.823160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] The token cache is being invalidate because project c36a3d54680846eea8ab292610a1d8c7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:48.823160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.823765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:48.825593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 973/1930] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:48 2022] DELETE /identity/v3/projects/c36a3d54680846eea8ab292610a1d8c7 => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:48.829216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:48.830456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:48.842414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ef73eee-baba-4a20-82c4-4e341b6fbac9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:48.847174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:48.848039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.848327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.848604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] PATH_INFO: `/v3/users/c2b9dab177c84b53b51a9999f4adf543` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:48.849874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2b9dab177c84b53b51a9999f4adf543)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:48.851796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:48.961368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 958/1931] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:48 2022] PUT /identity/v3/projects/0d771f49f6b641998dc3bc54229534bb/users/d7a57249572345429ba6e706b22a7c5d/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:48.972025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3f7f17-0442-4429-9655-20572b76951b None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:48.972323 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3f7f17-0442-4429-9655-20572b76951b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:48.972612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c3f7f17-0442-4429-9655-20572b76951b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:49.000547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9c3f7f17-0442-4429-9655-20572b76951b None None] MFA Rules not processed for user `d7a57249572345429ba6e706b22a7c5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:49.001793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9c3f7f17-0442-4429-9655-20572b76951b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:49.068599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 959/1932] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:48 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:49.080650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3f7f17-0442-4429-9655-20572b76951b None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:49.082029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9c3f7f17-0442-4429-9655-20572b76951b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:49.145873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c3f7f17-0442-4429-9655-20572b76951b None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:49.150604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecbc8d6-696d-4e11-b522-45655af53a5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:49.151287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecbc8d6-696d-4e11-b522-45655af53a5a admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:49.151613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecbc8d6-696d-4e11-b522-45655af53a5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:49.151890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecbc8d6-696d-4e11-b522-45655af53a5a admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:49.153158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecbc8d6-696d-4e11-b522-45655af53a5a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:49.155093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecbc8d6-696d-4e11-b522-45655af53a5a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:49.180102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2b9dab177c84b53b51a9999f4adf543'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:49.180751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2b9dab177c84b53b51a9999f4adf543'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:49.187876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2b9dab177c84b53b51a9999f4adf543'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:49.326227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 960/1933] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:49 2022] POST /identity/v3/projects => generated 379 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:05:49.327169 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecbc8d6-696d-4e11-b522-45655af53a5a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:49.328978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fecbc8d6-696d-4e11-b522-45655af53a5a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:49.372105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2b9dab177c84b53b51a9999f4adf543'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:49.372828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2b9dab177c84b53b51a9999f4adf543'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:49.391681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 974/1934] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:48 2022] DELETE /identity/v3/users/c2b9dab177c84b53b51a9999f4adf543 => generated 0 bytes in 563 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:49.397372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecbc8d6-696d-4e11-b522-45655af53a5a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:49.453431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6ea0d-3969-4a02-a6d4-85e3205512ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:49.454115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6ea0d-3969-4a02-a6d4-85e3205512ca admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:49.454421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6ea0d-3969-4a02-a6d4-85e3205512ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:49.454700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6ea0d-3969-4a02-a6d4-85e3205512ca admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:49.455380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d6ea0d-3969-4a02-a6d4-85e3205512ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:49.457334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d6ea0d-3969-4a02-a6d4-85e3205512ca admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:49.634758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 961/1935] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:49 2022] POST /identity/v3/users => generated 296 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:49.639917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:49.651379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed778369-2ce7-46da-b458-e0b0d07eac4d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:49.656797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7501f7-544d-48ef-9cea-833b24bcf5bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:49.657553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7501f7-544d-48ef-9cea-833b24bcf5bb admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:49.657832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7501f7-544d-48ef-9cea-833b24bcf5bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:49.658104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7501f7-544d-48ef-9cea-833b24bcf5bb admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:49.658832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7501f7-544d-48ef-9cea-833b24bcf5bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:49.660430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7501f7-544d-48ef-9cea-833b24bcf5bb admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:49.672346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 975/1936] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:49 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:05:49.676111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6ea0d-3969-4a02-a6d4-85e3205512ca admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:49.686532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6ea0d-3969-4a02-a6d4-85e3205512ca admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:49.691602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d837a3-65a3-4567-bb7e-1b75f11bb3d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:49.692530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d837a3-65a3-4567-bb7e-1b75f11bb3d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:49.692841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d837a3-65a3-4567-bb7e-1b75f11bb3d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:49.693168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d837a3-65a3-4567-bb7e-1b75f11bb3d1 admin admin] PATH_INFO: `/v3/projects/76dda784c1534d699be1e013165627a3/users/f1ae3414b1404aaca11e385510e688d1/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:49.714242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d837a3-65a3-4567-bb7e-1b75f11bb3d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76dda784c1534d699be1e013165627a3, user_id=f1ae3414b1404aaca11e385510e688d1, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:49.718365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d837a3-65a3-4567-bb7e-1b75f11bb3d1 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:49.720997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:49.808711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 962/1937] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:49 2022] PUT /identity/v3/projects/76dda784c1534d699be1e013165627a3/users/f1ae3414b1404aaca11e385510e688d1/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:49.812717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7501f7-544d-48ef-9cea-833b24bcf5bb admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:49.827959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7501f7-544d-48ef-9cea-833b24bcf5bb admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:49.918635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05df6c66-2ef5-41d7-8be7-e016f2da1737 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:49.919123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05df6c66-2ef5-41d7-8be7-e016f2da1737 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:49.919428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05df6c66-2ef5-41d7-8be7-e016f2da1737 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:49.919720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05df6c66-2ef5-41d7-8be7-e016f2da1737 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:49.920439 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05df6c66-2ef5-41d7-8be7-e016f2da1737 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:49.922126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05df6c66-2ef5-41d7-8be7-e016f2da1737 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:49.936149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 976/1938] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:49 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:49.937386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d837a3-65a3-4567-bb7e-1b75f11bb3d1 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:49.949133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d837a3-65a3-4567-bb7e-1b75f11bb3d1 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:49.957558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a16bee-cfeb-43a0-96c4-a0ab1072e5e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:49.958610 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a16bee-cfeb-43a0-96c4-a0ab1072e5e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:49.958926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a16bee-cfeb-43a0-96c4-a0ab1072e5e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:49.959207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a16bee-cfeb-43a0-96c4-a0ab1072e5e5 admin admin] PATH_INFO: `/v3/projects/76dda784c1534d699be1e013165627a3/users/f1ae3414b1404aaca11e385510e688d1/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:49.961302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a16bee-cfeb-43a0-96c4-a0ab1072e5e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=76dda784c1534d699be1e013165627a3, user_id=f1ae3414b1404aaca11e385510e688d1, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:49.964203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a16bee-cfeb-43a0-96c4-a0ab1072e5e5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:49.975134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 963/1939] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:49 2022] PUT /identity/v3/projects/76dda784c1534d699be1e013165627a3/users/f1ae3414b1404aaca11e385510e688d1/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:49.990093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceec075-102c-4f0d-b3fc-2246bfe9fdd0 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:49.990733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceec075-102c-4f0d-b3fc-2246bfe9fdd0 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:49.991020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ceec075-102c-4f0d-b3fc-2246bfe9fdd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.019430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4ceec075-102c-4f0d-b3fc-2246bfe9fdd0 None None] MFA Rules not processed for user `f1ae3414b1404aaca11e385510e688d1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:50.020370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a16bee-cfeb-43a0-96c4-a0ab1072e5e5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:50.020978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4ceec075-102c-4f0d-b3fc-2246bfe9fdd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:50.022346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-17a16bee-cfeb-43a0-96c4-a0ab1072e5e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:50.100332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a16bee-cfeb-43a0-96c4-a0ab1072e5e5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:50.105614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:50.106636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:50.107066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:50.107320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] PATH_INFO: `/v3/projects/3d39d69fbaa24f6e8b6e6af8f2198733` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.108853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] RBAC: Authorizing `identity:delete_project(project_id=3d39d69fbaa24f6e8b6e6af8f2198733)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:50.110825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:50.113082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 977/1940] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:49 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:50.122544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceec075-102c-4f0d-b3fc-2246bfe9fdd0 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:50.124103 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4ceec075-102c-4f0d-b3fc-2246bfe9fdd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:50.196174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ceec075-102c-4f0d-b3fc-2246bfe9fdd0 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:50.202200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac8bcd94-dd21-4981-b5cf-6cea394ce40c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:50.203083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8bcd94-dd21-4981-b5cf-6cea394ce40c service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:50.203558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8bcd94-dd21-4981-b5cf-6cea394ce40c service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:50.204006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac8bcd94-dd21-4981-b5cf-6cea394ce40c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.221153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8bcd94-dd21-4981-b5cf-6cea394ce40c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:50.222618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac8bcd94-dd21-4981-b5cf-6cea394ce40c service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:50.243615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 978/1941] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:50 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:05:50.285622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3d39d69fbaa24f6e8b6e6af8f2198733', 'request_id': 'req-2123ffb8-2911-4573-99d8-3768898e779b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:50.418411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3d39d69fbaa24f6e8b6e6af8f2198733', 'request_id': 'req-2123ffb8-2911-4573-99d8-3768898e779b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:50.425839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3d39d69fbaa24f6e8b6e6af8f2198733', 'request_id': 'req-2123ffb8-2911-4573-99d8-3768898e779b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:50.438047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] The token cache is being invalidate because project 3d39d69fbaa24f6e8b6e6af8f2198733 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:50.445063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:50.445063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2123ffb8-2911-4573-99d8-3768898e779b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:50.445063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 964/1942] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:50 2022] DELETE /identity/v3/projects/3d39d69fbaa24f6e8b6e6af8f2198733 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:50.446513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb346e-2247-4015-91c8-c1e4bc561803 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:50.446812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb346e-2247-4015-91c8-c1e4bc561803 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:50.447094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bdb346e-2247-4015-91c8-c1e4bc561803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.465539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-7bdb346e-2247-4015-91c8-c1e4bc561803 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:50.466735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7bdb346e-2247-4015-91c8-c1e4bc561803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:50.529087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 979/1943] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:50.537302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4625c250-695d-461f-9259-fa6e18679ead None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:50.537602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4625c250-695d-461f-9259-fa6e18679ead None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:50.537893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4625c250-695d-461f-9259-fa6e18679ead None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.557697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-4625c250-695d-461f-9259-fa6e18679ead None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:50.559508 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4625c250-695d-461f-9259-fa6e18679ead None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:50.568468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 965/1944] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:50.572454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdb346e-2247-4015-91c8-c1e4bc561803 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:50.584223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bdb346e-2247-4015-91c8-c1e4bc561803 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:50.589147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66f175b-8f03-430f-94e3-51c055670b24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:50.590331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66f175b-8f03-430f-94e3-51c055670b24 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:50.590614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66f175b-8f03-430f-94e3-51c055670b24 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:50.590908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66f175b-8f03-430f-94e3-51c055670b24 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.591712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66f175b-8f03-430f-94e3-51c055670b24 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:50.591895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66f175b-8f03-430f-94e3-51c055670b24 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:50.592674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:50.593546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66f175b-8f03-430f-94e3-51c055670b24 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:50.621750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 980/1945] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:50.622863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4625c250-695d-461f-9259-fa6e18679ead None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:50.632561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4625c250-695d-461f-9259-fa6e18679ead None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:50.639158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dc39fa-33fc-4448-b810-70b179fd7058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:50.639717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dc39fa-33fc-4448-b810-70b179fd7058 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:50.640074 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dc39fa-33fc-4448-b810-70b179fd7058 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:50.640892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dc39fa-33fc-4448-b810-70b179fd7058 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.641119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94dc39fa-33fc-4448-b810-70b179fd7058 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:50.643030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94dc39fa-33fc-4448-b810-70b179fd7058 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:50.736865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 966/1946] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:50 2022] POST /identity/v3/projects => generated 377 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:50.738819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66f175b-8f03-430f-94e3-51c055670b24 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:50.749134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66f175b-8f03-430f-94e3-51c055670b24 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:50.809072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e0a81e-f831-4c0b-bbc2-324df198c72a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:50.809753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e0a81e-f831-4c0b-bbc2-324df198c72a admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:50.810098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e0a81e-f831-4c0b-bbc2-324df198c72a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:50.810546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e0a81e-f831-4c0b-bbc2-324df198c72a admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.811253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e0a81e-f831-4c0b-bbc2-324df198c72a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:50.813344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e0a81e-f831-4c0b-bbc2-324df198c72a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:50.929729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dc39fa-33fc-4448-b810-70b179fd7058 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:50.931574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-94dc39fa-33fc-4448-b810-70b179fd7058 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:50.943986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dc39fa-33fc-4448-b810-70b179fd7058 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:50.948664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:50.950159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:50.950445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:50.950780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] PATH_INFO: `/v3/users/d7a57249572345429ba6e706b22a7c5d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.951958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7a57249572345429ba6e706b22a7c5d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:50.953949 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:50.959655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 981/1947] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:50 2022] POST /identity/v3/users => generated 289 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:50.960907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e0a81e-f831-4c0b-bbc2-324df198c72a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:50.976975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e0a81e-f831-4c0b-bbc2-324df198c72a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:50.982087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3136896d-1cc9-4b09-a57a-0b688a41bd30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:50.982790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3136896d-1cc9-4b09-a57a-0b688a41bd30 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:50.983077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3136896d-1cc9-4b09-a57a-0b688a41bd30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:50.983361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3136896d-1cc9-4b09-a57a-0b688a41bd30 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:50.984089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3136896d-1cc9-4b09-a57a-0b688a41bd30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:50.985723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3136896d-1cc9-4b09-a57a-0b688a41bd30 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:51.000117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 982/1948] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:50 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:51.001319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3136896d-1cc9-4b09-a57a-0b688a41bd30 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:51.011117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3136896d-1cc9-4b09-a57a-0b688a41bd30 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:51.017983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a771b33-816e-40b9-9a7b-0a355525dfd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:51.018650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a771b33-816e-40b9-9a7b-0a355525dfd9 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:51.018939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a771b33-816e-40b9-9a7b-0a355525dfd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:51.019222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a771b33-816e-40b9-9a7b-0a355525dfd9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:51.019949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a771b33-816e-40b9-9a7b-0a355525dfd9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:51.021816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a771b33-816e-40b9-9a7b-0a355525dfd9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:51.032498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 983/1949] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:51 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:05:51.036823 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a771b33-816e-40b9-9a7b-0a355525dfd9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:51.046137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a771b33-816e-40b9-9a7b-0a355525dfd9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:51.050731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b6bda3d-52ff-4480-8c79-27d06044c682 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:51.051576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6bda3d-52ff-4480-8c79-27d06044c682 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:51.051861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6bda3d-52ff-4480-8c79-27d06044c682 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:51.052142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b6bda3d-52ff-4480-8c79-27d06044c682 admin admin] PATH_INFO: `/v3/projects/a40832ec7fa34f71b4657fc372672f41/users/9b0e50775367405c8d997036dd33aac7/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:51.066594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6bda3d-52ff-4480-8c79-27d06044c682 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a40832ec7fa34f71b4657fc372672f41, user_id=9b0e50775367405c8d997036dd33aac7, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:51.069411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b6bda3d-52ff-4480-8c79-27d06044c682 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:51.214271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7a57249572345429ba6e706b22a7c5d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:51.215461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7a57249572345429ba6e706b22a7c5d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:51.216068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7a57249572345429ba6e706b22a7c5d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:51.222866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7a57249572345429ba6e706b22a7c5d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:51.252994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 984/1950] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:51 2022] PUT /identity/v3/projects/a40832ec7fa34f71b4657fc372672f41/users/9b0e50775367405c8d997036dd33aac7/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:51.264211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9d4a8f-bfbb-49f8-9959-d160497bdd1c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:51.264508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9d4a8f-bfbb-49f8-9959-d160497bdd1c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:51.264817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9d4a8f-bfbb-49f8-9959-d160497bdd1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:51.291009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3b9d4a8f-bfbb-49f8-9959-d160497bdd1c None None] MFA Rules not processed for user `9b0e50775367405c8d997036dd33aac7`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:51.292276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3b9d4a8f-bfbb-49f8-9959-d160497bdd1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:51.309507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c720d25f-c5f6-4a67-8ee8-6b948b2f3684 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7a57249572345429ba6e706b22a7c5d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:51.329525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 967/1951] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:50 2022] DELETE /identity/v3/users/d7a57249572345429ba6e706b22a7c5d => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:51.340157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd2481-9738-4c2d-af47-1a66e301cec9 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:51.340465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd2481-9738-4c2d-af47-1a66e301cec9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:51.340748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7dd2481-9738-4c2d-af47-1a66e301cec9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:51.361526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a7dd2481-9738-4c2d-af47-1a66e301cec9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:51.363045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a7dd2481-9738-4c2d-af47-1a66e301cec9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:51.369565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 985/1952] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:51 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:51.386261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9d4a8f-bfbb-49f8-9959-d160497bdd1c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:51.387746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3b9d4a8f-bfbb-49f8-9959-d160497bdd1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:51.430031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 968/1953] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:51.439675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd2481-9738-4c2d-af47-1a66e301cec9 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:51.441235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a7dd2481-9738-4c2d-af47-1a66e301cec9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:51.445638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9d4a8f-bfbb-49f8-9959-d160497bdd1c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:51.491978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=94,keystone:DELETE=4,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:51.501704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7dd2481-9738-4c2d-af47-1a66e301cec9 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:51.503488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f46564a-957f-4ea4-a935-44e19a7a0231 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:51.503488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f46564a-957f-4ea4-a935-44e19a7a0231 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:51.503488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f46564a-957f-4ea4-a935-44e19a7a0231 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:51.503924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f46564a-957f-4ea4-a935-44e19a7a0231 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:51.505446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb04729-079b-42b9-938e-eda5b6e16f01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:51.506097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb04729-079b-42b9-938e-eda5b6e16f01 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:51.506384 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb04729-079b-42b9-938e-eda5b6e16f01 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:51.506663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb04729-079b-42b9-938e-eda5b6e16f01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:51.515712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb04729-079b-42b9-938e-eda5b6e16f01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:51.517206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb04729-079b-42b9-938e-eda5b6e16f01 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:51.526824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 969/1954] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:51.562384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f46564a-957f-4ea4-a935-44e19a7a0231 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:51.563844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f46564a-957f-4ea4-a935-44e19a7a0231 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:51.574419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 986/1955] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:51 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:05:51.956436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb04729-079b-42b9-938e-eda5b6e16f01 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:51.957873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7fb04729-079b-42b9-938e-eda5b6e16f01 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:51.971505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb04729-079b-42b9-938e-eda5b6e16f01 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:51.976236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:51.977135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:51.977474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:51.977702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] PATH_INFO: `/v3/projects/0d771f49f6b641998dc3bc54229534bb` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:51.978887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d771f49f6b641998dc3bc54229534bb)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:51.980858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:52.105725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d771f49f6b641998dc3bc54229534bb', 'request_id': 'req-2c010fdc-3413-466d-8633-a068e9a06920'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.309904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:52.545075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d771f49f6b641998dc3bc54229534bb', 'request_id': 'req-2c010fdc-3413-466d-8633-a068e9a06920'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.551919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d771f49f6b641998dc3bc54229534bb', 'request_id': 'req-2c010fdc-3413-466d-8633-a068e9a06920'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.564794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] The token cache is being invalidate because project 0d771f49f6b641998dc3bc54229534bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:52.565200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.565881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.567991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 970/1956] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:51 2022] DELETE /identity/v3/projects/0d771f49f6b641998dc3bc54229534bb => generated 0 bytes in 612 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:52.572183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f46564a-957f-4ea4-a935-44e19a7a0231 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:52.584950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f46564a-957f-4ea4-a935-44e19a7a0231 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:52.642780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:52.643645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:52.643928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:52.644205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] PATH_INFO: `/v3/users/f1ae3414b1404aaca11e385510e688d1` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:52.645472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] RBAC: Authorizing `identity:delete_user(user_id=f1ae3414b1404aaca11e385510e688d1)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:52.647354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:52.836987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f1ae3414b1404aaca11e385510e688d1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.837586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f1ae3414b1404aaca11e385510e688d1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.845146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f1ae3414b1404aaca11e385510e688d1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.937119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f1ae3414b1404aaca11e385510e688d1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.937952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-3e67f6bd-7119-4cb7-a4fe-74405abbca5d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f1ae3414b1404aaca11e385510e688d1'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:52.959106 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 987/1957] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:52 2022] DELETE /identity/v3/users/f1ae3414b1404aaca11e385510e688d1 => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:53.526312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:53.527899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:53.543294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c010fdc-3413-466d-8633-a068e9a06920 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:53.549215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:53.627346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:53.628213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:53.628500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:53.628809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] PATH_INFO: `/v3/projects/76dda784c1534d699be1e013165627a3` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:53.630004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] RBAC: Authorizing `identity:delete_project(project_id=76dda784c1534d699be1e013165627a3)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:53.631952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:53.674731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=87,keystone:DELETE=5,keystone:INSERT=8 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:53.751799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '76dda784c1534d699be1e013165627a3', 'request_id': 'req-2be4e19e-a479-4421-9cf0-fdca6e9ad711'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:53.837302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '76dda784c1534d699be1e013165627a3', 'request_id': 'req-2be4e19e-a479-4421-9cf0-fdca6e9ad711'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:53.846421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '76dda784c1534d699be1e013165627a3', 'request_id': 'req-2be4e19e-a479-4421-9cf0-fdca6e9ad711'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:53.858397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] The token cache is being invalidate because project 76dda784c1534d699be1e013165627a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:53.858691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:53.859381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2be4e19e-a479-4421-9cf0-fdca6e9ad711 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:53.861178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 971/1958] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:53 2022] DELETE /identity/v3/projects/76dda784c1534d699be1e013165627a3 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:53.866160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe77271-a439-47e4-8a6f-c086fe8b8269 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:53.866618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe77271-a439-47e4-8a6f-c086fe8b8269 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:53.866733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe77271-a439-47e4-8a6f-c086fe8b8269 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:53.885228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6fe77271-a439-47e4-8a6f-c086fe8b8269 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:53.886555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6fe77271-a439-47e4-8a6f-c086fe8b8269 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:53.953649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 988/1959] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:53.961868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc06890c-d716-4519-908c-96575e0a8457 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:53.962139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc06890c-d716-4519-908c-96575e0a8457 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:53.962421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc06890c-d716-4519-908c-96575e0a8457 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:53.981495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-bc06890c-d716-4519-908c-96575e0a8457 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:53.982719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bc06890c-d716-4519-908c-96575e0a8457 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:53.991891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 972/1960] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:53.996915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe77271-a439-47e4-8a6f-c086fe8b8269 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:54.007412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe77271-a439-47e4-8a6f-c086fe8b8269 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:54.013149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:54.013819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:54.017170 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:54.017170 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:54.017170 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:54.017170 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:54.017451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:54.029860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 989/1961] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:54.037202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc06890c-d716-4519-908c-96575e0a8457 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:54.048009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc06890c-d716-4519-908c-96575e0a8457 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:54.052806 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0748602f-14f5-483e-84dc-12b44bc207a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:54.053483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0748602f-14f5-483e-84dc-12b44bc207a9 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:54.053782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0748602f-14f5-483e-84dc-12b44bc207a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:54.054050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0748602f-14f5-483e-84dc-12b44bc207a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:54.054743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0748602f-14f5-483e-84dc-12b44bc207a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:54.056741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0748602f-14f5-483e-84dc-12b44bc207a9 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:54.150989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:54.157120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:54.195861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 973/1962] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:54 2022] POST /identity/v3/projects => generated 385 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:54.196716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0748602f-14f5-483e-84dc-12b44bc207a9 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:54.232256 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5131c123-3faa-42a1-9d2e-88e6f1542269 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:54.293520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0748602f-14f5-483e-84dc-12b44bc207a9 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:54.299333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf1ef7-3d03-41e6-b3f8-cc82968ab855 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:54.308189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf1ef7-3d03-41e6-b3f8-cc82968ab855 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:54.308189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf1ef7-3d03-41e6-b3f8-cc82968ab855 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:54.308189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bbf1ef7-3d03-41e6-b3f8-cc82968ab855 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:54.308189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbf1ef7-3d03-41e6-b3f8-cc82968ab855 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:54.308189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bbf1ef7-3d03-41e6-b3f8-cc82968ab855 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:54.308853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec848531-9186-4269-8194-2f840033aff9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:54.308853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec848531-9186-4269-8194-2f840033aff9 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:54.308853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec848531-9186-4269-8194-2f840033aff9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:54.308853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec848531-9186-4269-8194-2f840033aff9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:54.309159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ec848531-9186-4269-8194-2f840033aff9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:54.395930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec848531-9186-4269-8194-2f840033aff9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:54.399608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec848531-9186-4269-8194-2f840033aff9 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:54.416466 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 990/1963] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:54 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:54.430251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 974/1964] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:54 2022] POST /identity/v3/users => generated 293 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:54.434782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec848531-9186-4269-8194-2f840033aff9 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:54.446010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec848531-9186-4269-8194-2f840033aff9 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:54.457260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7da173-70fc-4d45-b330-c8c5e195964e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:54.457953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7da173-70fc-4d45-b330-c8c5e195964e admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:54.458243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7da173-70fc-4d45-b330-c8c5e195964e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:54.458527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7da173-70fc-4d45-b330-c8c5e195964e admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:54.459269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7da173-70fc-4d45-b330-c8c5e195964e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:54.465722 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7da173-70fc-4d45-b330-c8c5e195964e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:54.480953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 991/1965] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:54 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:54.482866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf1ef7-3d03-41e6-b3f8-cc82968ab855 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:54.498850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bbf1ef7-3d03-41e6-b3f8-cc82968ab855 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:54.505495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fec4924-ede5-4937-bbc5-307fba7277e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:54.506372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec4924-ede5-4937-bbc5-307fba7277e1 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:54.506646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec4924-ede5-4937-bbc5-307fba7277e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:54.506925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fec4924-ede5-4937-bbc5-307fba7277e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:54.507902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fec4924-ede5-4937-bbc5-307fba7277e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:54.509365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fec4924-ede5-4937-bbc5-307fba7277e1 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:54.524387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 975/1966] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:54 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:54.525611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7da173-70fc-4d45-b330-c8c5e195964e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:54.536424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7da173-70fc-4d45-b330-c8c5e195964e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:54.541590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dd4d0f-3a75-486a-898e-66f1d3406582 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:54.543051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dd4d0f-3a75-486a-898e-66f1d3406582 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:54.543347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dd4d0f-3a75-486a-898e-66f1d3406582 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:54.543630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32dd4d0f-3a75-486a-898e-66f1d3406582 admin admin] PATH_INFO: `/v3/projects/c542fc3d32b8464a944116d6beffc8bf/users/624700c503f5495dbb215d23557c6968/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:54.565502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dd4d0f-3a75-486a-898e-66f1d3406582 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c542fc3d32b8464a944116d6beffc8bf, user_id=624700c503f5495dbb215d23557c6968, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:54.568573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32dd4d0f-3a75-486a-898e-66f1d3406582 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:54.661751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 992/1967] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:54 2022] PUT /identity/v3/projects/c542fc3d32b8464a944116d6beffc8bf/users/624700c503f5495dbb215d23557c6968/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:54.674778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55631468-5625-4ff5-843b-006a5a72c37c None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:54.675333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55631468-5625-4ff5-843b-006a5a72c37c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:54.675333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55631468-5625-4ff5-843b-006a5a72c37c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:54.702742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-55631468-5625-4ff5-843b-006a5a72c37c None None] MFA Rules not processed for user `624700c503f5495dbb215d23557c6968`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:54.703971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-55631468-5625-4ff5-843b-006a5a72c37c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:54.775599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 976/1968] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:54 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:54.780687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dd4d0f-3a75-486a-898e-66f1d3406582 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:54.848275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32dd4d0f-3a75-486a-898e-66f1d3406582 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:54.854230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee067644-34d9-4e85-a26a-c7b88352ca8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:54.854977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee067644-34d9-4e85-a26a-c7b88352ca8b admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:54.855405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee067644-34d9-4e85-a26a-c7b88352ca8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:54.856010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee067644-34d9-4e85-a26a-c7b88352ca8b admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:54.856732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee067644-34d9-4e85-a26a-c7b88352ca8b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:54.859644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee067644-34d9-4e85-a26a-c7b88352ca8b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:54.994754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 993/1969] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:54 2022] POST /identity/v3/projects => generated 381 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:54.996689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55631468-5625-4ff5-843b-006a5a72c37c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:55.007725 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55631468-5625-4ff5-843b-006a5a72c37c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:55.064912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de46c5c7-d8f0-4a53-b240-e3282fe01477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:55.065597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de46c5c7-d8f0-4a53-b240-e3282fe01477 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.065888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de46c5c7-d8f0-4a53-b240-e3282fe01477 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.066167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de46c5c7-d8f0-4a53-b240-e3282fe01477 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.066862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de46c5c7-d8f0-4a53-b240-e3282fe01477 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:55.069358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de46c5c7-d8f0-4a53-b240-e3282fe01477 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:55.218968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 977/1970] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:54 2022] POST /identity/v3/users => generated 291 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:55.231212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee067644-34d9-4e85-a26a-c7b88352ca8b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:55.239292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee067644-34d9-4e85-a26a-c7b88352ca8b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:55.244335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0e4f74-567c-4e16-b97d-8e668b830cff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:55.245202 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0e4f74-567c-4e16-b97d-8e668b830cff admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.246079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0e4f74-567c-4e16-b97d-8e668b830cff admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.246332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0e4f74-567c-4e16-b97d-8e668b830cff admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.247134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0e4f74-567c-4e16-b97d-8e668b830cff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:55.252723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0e4f74-567c-4e16-b97d-8e668b830cff admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:55.269674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de46c5c7-d8f0-4a53-b240-e3282fe01477 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:55.274842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 994/1971] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:55 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:55.281823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de46c5c7-d8f0-4a53-b240-e3282fe01477 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:55.287028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0735dd7c-ef58-4380-8c48-e715115f1e3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:55.287997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0735dd7c-ef58-4380-8c48-e715115f1e3b admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.288326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0735dd7c-ef58-4380-8c48-e715115f1e3b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.288617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0735dd7c-ef58-4380-8c48-e715115f1e3b admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.289381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0735dd7c-ef58-4380-8c48-e715115f1e3b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:55.291002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0735dd7c-ef58-4380-8c48-e715115f1e3b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:55.301817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 978/1972] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:55 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:55.311944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0e4f74-567c-4e16-b97d-8e668b830cff admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:55.313859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0735dd7c-ef58-4380-8c48-e715115f1e3b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:55.315555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0735dd7c-ef58-4380-8c48-e715115f1e3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:55.322676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0e4f74-567c-4e16-b97d-8e668b830cff admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:55.329168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c148673-7ad8-49a1-8a0f-c4016e5ee985 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:55.329654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c148673-7ad8-49a1-8a0f-c4016e5ee985 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.330056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c148673-7ad8-49a1-8a0f-c4016e5ee985 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.330346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c148673-7ad8-49a1-8a0f-c4016e5ee985 admin admin] PATH_INFO: `/v3/projects/7f2b65d4a7ab4359983515c792bf42d3/users/d502709541734ea0afb87adec4dbe771/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.331938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0735dd7c-ef58-4380-8c48-e715115f1e3b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:55.337851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:55.338829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.339113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.339443 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] PATH_INFO: `/v3/users/53d8c34a5e14484eb0366b0f6de4ebcd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.340857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=53d8c34a5e14484eb0366b0f6de4ebcd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:55.343214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:55.347435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c148673-7ad8-49a1-8a0f-c4016e5ee985 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f2b65d4a7ab4359983515c792bf42d3, user_id=d502709541734ea0afb87adec4dbe771, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:55.351013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c148673-7ad8-49a1-8a0f-c4016e5ee985 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:55.418956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 995/1973] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:55 2022] PUT /identity/v3/projects/7f2b65d4a7ab4359983515c792bf42d3/users/d502709541734ea0afb87adec4dbe771/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:55.429921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54915d88-8803-4a98-95e2-aaef7ca5e45b None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.430291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54915d88-8803-4a98-95e2-aaef7ca5e45b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.430638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54915d88-8803-4a98-95e2-aaef7ca5e45b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.459500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-54915d88-8803-4a98-95e2-aaef7ca5e45b None None] MFA Rules not processed for user `d502709541734ea0afb87adec4dbe771`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:55.460804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-54915d88-8803-4a98-95e2-aaef7ca5e45b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:55.485413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '53d8c34a5e14484eb0366b0f6de4ebcd'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:55.486304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '53d8c34a5e14484eb0366b0f6de4ebcd'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:55.487080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '53d8c34a5e14484eb0366b0f6de4ebcd'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:55.509135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '53d8c34a5e14484eb0366b0f6de4ebcd'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:55.515142 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:55.537682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:55.580708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 996/1974] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:55 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:05:55.580708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54915d88-8803-4a98-95e2-aaef7ca5e45b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:55.583490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-54915d88-8803-4a98-95e2-aaef7ca5e45b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:55.600307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8fb9ab7c-f87d-4087-8c60-9ab90ff087cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '53d8c34a5e14484eb0366b0f6de4ebcd'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:55.619956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 979/1975] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:55 2022] DELETE /identity/v3/users/53d8c34a5e14484eb0366b0f6de4ebcd => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:55.625203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbacfc0-02b7-4f3b-b9ce-c40b1cc17a02 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.625203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbacfc0-02b7-4f3b-b9ce-c40b1cc17a02 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.625515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbacfc0-02b7-4f3b-b9ce-c40b1cc17a02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.642793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-fdbacfc0-02b7-4f3b-b9ce-c40b1cc17a02 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:55.644197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fdbacfc0-02b7-4f3b-b9ce-c40b1cc17a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:55.660344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54915d88-8803-4a98-95e2-aaef7ca5e45b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:55.712531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 980/1976] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:55.722262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbacfc0-02b7-4f3b-b9ce-c40b1cc17a02 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:55.723835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fdbacfc0-02b7-4f3b-b9ce-c40b1cc17a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:55.727779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31463fa5-c860-4a71-86c8-734363575b6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:55.733475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31463fa5-c860-4a71-86c8-734363575b6b admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.733475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31463fa5-c860-4a71-86c8-734363575b6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.733475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31463fa5-c860-4a71-86c8-734363575b6b admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.733475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31463fa5-c860-4a71-86c8-734363575b6b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:55.733475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31463fa5-c860-4a71-86c8-734363575b6b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:55.784037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbacfc0-02b7-4f3b-b9ce-c40b1cc17a02 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:55.787811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13460555-82a5-4093-b9eb-c0f9d4a353d9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:55.788464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13460555-82a5-4093-b9eb-c0f9d4a353d9 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.788751 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13460555-82a5-4093-b9eb-c0f9d4a353d9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.789082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13460555-82a5-4093-b9eb-c0f9d4a353d9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.800965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13460555-82a5-4093-b9eb-c0f9d4a353d9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:55.802463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13460555-82a5-4093-b9eb-c0f9d4a353d9 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:55.813264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 981/1977] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:05:55.833971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 997/1978] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:55 2022] POST /identity/v3/projects => generated 385 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:55.835064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13460555-82a5-4093-b9eb-c0f9d4a353d9 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:55.845477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13460555-82a5-4093-b9eb-c0f9d4a353d9 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:55.914059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9432a2-9f06-4a2d-b028-f86cbed924b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:55.915531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9432a2-9f06-4a2d-b028-f86cbed924b0 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:55.916316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9432a2-9f06-4a2d-b028-f86cbed924b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:55.916979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9432a2-9f06-4a2d-b028-f86cbed924b0 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:55.919824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9432a2-9f06-4a2d-b028-f86cbed924b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:55.921835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9432a2-9f06-4a2d-b028-f86cbed924b0 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:56.017724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 982/1979] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:55 2022] POST /identity/v3/users => generated 299 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:56.020281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31463fa5-c860-4a71-86c8-734363575b6b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:56.031301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31463fa5-c860-4a71-86c8-734363575b6b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:56.036449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e871ff4-0a68-4365-b542-a0c3f98b6652 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:56.040626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e871ff4-0a68-4365-b542-a0c3f98b6652 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:56.040626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e871ff4-0a68-4365-b542-a0c3f98b6652 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:56.040626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e871ff4-0a68-4365-b542-a0c3f98b6652 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:56.040626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e871ff4-0a68-4365-b542-a0c3f98b6652 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:56.040626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e871ff4-0a68-4365-b542-a0c3f98b6652 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:56.051673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 998/1980] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:56 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:56.056071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9432a2-9f06-4a2d-b028-f86cbed924b0 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:56.066111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9432a2-9f06-4a2d-b028-f86cbed924b0 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:56.070981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67da2d8a-431f-4617-b1ad-6e5a4c5554bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:56.071864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67da2d8a-431f-4617-b1ad-6e5a4c5554bc admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:56.072151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67da2d8a-431f-4617-b1ad-6e5a4c5554bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:56.072432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67da2d8a-431f-4617-b1ad-6e5a4c5554bc admin admin] PATH_INFO: `/v3/projects/7012d2f7c89a4bf2afc4e500980d6223/users/fb3ee230dcec4ff79d073321f003491e/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:56.088974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67da2d8a-431f-4617-b1ad-6e5a4c5554bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=7012d2f7c89a4bf2afc4e500980d6223, user_id=fb3ee230dcec4ff79d073321f003491e, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:56.092549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67da2d8a-431f-4617-b1ad-6e5a4c5554bc admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:56.201471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 983/1981] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:56 2022] PUT /identity/v3/projects/7012d2f7c89a4bf2afc4e500980d6223/users/fb3ee230dcec4ff79d073321f003491e/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:56.205689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e871ff4-0a68-4365-b542-a0c3f98b6652 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:56.217990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e871ff4-0a68-4365-b542-a0c3f98b6652 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:56.291109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865b0474-1ec7-4cb2-8c1e-8b123e90ebf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:56.291797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865b0474-1ec7-4cb2-8c1e-8b123e90ebf0 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:56.292114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865b0474-1ec7-4cb2-8c1e-8b123e90ebf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:56.292416 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865b0474-1ec7-4cb2-8c1e-8b123e90ebf0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:56.293213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865b0474-1ec7-4cb2-8c1e-8b123e90ebf0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:56.294990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865b0474-1ec7-4cb2-8c1e-8b123e90ebf0 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:56.308640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 999/1982] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:56 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:05:56.309875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67da2d8a-431f-4617-b1ad-6e5a4c5554bc admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:56.320556 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67da2d8a-431f-4617-b1ad-6e5a4c5554bc admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:56.326050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70241fd-380f-46ec-aea7-143d7d656f18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:56.326938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70241fd-380f-46ec-aea7-143d7d656f18 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:56.327218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70241fd-380f-46ec-aea7-143d7d656f18 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:56.327808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70241fd-380f-46ec-aea7-143d7d656f18 admin admin] PATH_INFO: `/v3/projects/7012d2f7c89a4bf2afc4e500980d6223/users/fb3ee230dcec4ff79d073321f003491e/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:56.329513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70241fd-380f-46ec-aea7-143d7d656f18 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7012d2f7c89a4bf2afc4e500980d6223, user_id=fb3ee230dcec4ff79d073321f003491e, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:56.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70241fd-380f-46ec-aea7-143d7d656f18 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:56.345318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 984/1983] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:56 2022] PUT /identity/v3/projects/7012d2f7c89a4bf2afc4e500980d6223/users/fb3ee230dcec4ff79d073321f003491e/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:05:56.350500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865b0474-1ec7-4cb2-8c1e-8b123e90ebf0 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:56.352677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-865b0474-1ec7-4cb2-8c1e-8b123e90ebf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:56.365112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1789bb9-3a86-49d1-8280-e9a045dd5458 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:56.372311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1789bb9-3a86-49d1-8280-e9a045dd5458 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:56.372311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1789bb9-3a86-49d1-8280-e9a045dd5458 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:56.372729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865b0474-1ec7-4cb2-8c1e-8b123e90ebf0 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:56.397029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-c1789bb9-3a86-49d1-8280-e9a045dd5458 None None] MFA Rules not processed for user `fb3ee230dcec4ff79d073321f003491e`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:56.398258 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c1789bb9-3a86-49d1-8280-e9a045dd5458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:56.420287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:56.444150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=90,keystone:DELETE=8,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:56.444389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:56.444389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:56.444389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:56.444389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] PATH_INFO: `/v3/projects/f000c174b9184feab8726af9f46ac8c3` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:56.444725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] RBAC: Authorizing `identity:delete_project(project_id=f000c174b9184feab8726af9f46ac8c3)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:56.444725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:56.489711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 985/1984] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:56 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:56.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1789bb9-3a86-49d1-8280-e9a045dd5458 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:56.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c1789bb9-3a86-49d1-8280-e9a045dd5458 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:56.565149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1789bb9-3a86-49d1-8280-e9a045dd5458 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:56.568630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c710fa-380f-4cf5-abd4-b214a1989747 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:56.569330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c710fa-380f-4cf5-abd4-b214a1989747 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:56.569612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c710fa-380f-4cf5-abd4-b214a1989747 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:56.569885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c710fa-380f-4cf5-abd4-b214a1989747 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:56.571637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b0c710fa-380f-4cf5-abd4-b214a1989747 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:56.642214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f000c174b9184feab8726af9f46ac8c3', 'request_id': 'req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:56.654191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c710fa-380f-4cf5-abd4-b214a1989747 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:56.654191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c710fa-380f-4cf5-abd4-b214a1989747 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:56.660489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f000c174b9184feab8726af9f46ac8c3', 'request_id': 'req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:56.663286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 986/1985] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:56 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:05:56.686682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f000c174b9184feab8726af9f46ac8c3', 'request_id': 'req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:56.776523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] The token cache is being invalidate because project f000c174b9184feab8726af9f46ac8c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:56.777283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:56.777731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:56.779335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1000/1986] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:56 2022] DELETE /identity/v3/projects/f000c174b9184feab8726af9f46ac8c3 => generated 0 bytes in 429 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:56.783347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c710fa-380f-4cf5-abd4-b214a1989747 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:56.799211 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c710fa-380f-4cf5-abd4-b214a1989747 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:56.863371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:56.864274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:56.864559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:56.864864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] PATH_INFO: `/v3/users/5fd21861e41b4f43b74bacd53757f628` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:56.866059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] RBAC: Authorizing `identity:delete_user(user_id=5fd21861e41b4f43b74bacd53757f628)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:56.867983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:56.880101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:57.092929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5fd21861e41b4f43b74bacd53757f628'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:57.093837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5fd21861e41b4f43b74bacd53757f628'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:57.094625 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5fd21861e41b4f43b74bacd53757f628'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:57.102359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5fd21861e41b4f43b74bacd53757f628'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:57.131661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=84,keystone:DELETE=11,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:57.222699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-07595711-bf51-4c6e-a2b4-748308d1175f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5fd21861e41b4f43b74bacd53757f628'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:57.241777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 987/1987] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:05:56 2022] DELETE /identity/v3/users/5fd21861e41b4f43b74bacd53757f628 => generated 0 bytes in 459 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:57.258215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:57.634586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:57.871615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:57.874571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:57.887926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0005e4-cb8f-47f0-a17b-d914605c8c8a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:57.945982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:57.946845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:57.947257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:57.947489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] PATH_INFO: `/v3/projects/5280bec0e015484aa1bbb7092acc945e` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:57.948683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5280bec0e015484aa1bbb7092acc945e)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:57.949928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:57.954315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:58.052816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5280bec0e015484aa1bbb7092acc945e', 'request_id': 'req-73862966-3ff3-40dd-8b5b-5403a0603af6'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:58.059802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5280bec0e015484aa1bbb7092acc945e', 'request_id': 'req-73862966-3ff3-40dd-8b5b-5403a0603af6'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:58.065308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5280bec0e015484aa1bbb7092acc945e', 'request_id': 'req-73862966-3ff3-40dd-8b5b-5403a0603af6'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:58.182788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] The token cache is being invalidate because project 5280bec0e015484aa1bbb7092acc945e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:05:58.184182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:58.184182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-73862966-3ff3-40dd-8b5b-5403a0603af6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:05:58.185551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1001/1988] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:05:57 2022] DELETE /identity/v3/projects/5280bec0e015484aa1bbb7092acc945e => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:58.191254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afacc9b-fe42-4444-a2b0-fc36b828c982 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:58.191550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afacc9b-fe42-4444-a2b0-fc36b828c982 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:58.191856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afacc9b-fe42-4444-a2b0-fc36b828c982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:58.210251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9afacc9b-fe42-4444-a2b0-fc36b828c982 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:58.211460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9afacc9b-fe42-4444-a2b0-fc36b828c982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:58.287223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 988/1989] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:58.290223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fe7ca0-3984-40b5-b8ab-256a0470c455 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:58.293638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fe7ca0-3984-40b5-b8ab-256a0470c455 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:58.294128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fe7ca0-3984-40b5-b8ab-256a0470c455 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:58.318435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-89fe7ca0-3984-40b5-b8ab-256a0470c455 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:58.320420 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-89fe7ca0-3984-40b5-b8ab-256a0470c455 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:58.322212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:58.334882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1002/1990] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:58.338057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afacc9b-fe42-4444-a2b0-fc36b828c982 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:58.349799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afacc9b-fe42-4444-a2b0-fc36b828c982 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:58.355788 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0395e1e-fa8e-4931-987e-001a8d4cef5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:58.356471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0395e1e-fa8e-4931-987e-001a8d4cef5e admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:58.356758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0395e1e-fa8e-4931-987e-001a8d4cef5e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:58.357085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0395e1e-fa8e-4931-987e-001a8d4cef5e admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:58.357814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0395e1e-fa8e-4931-987e-001a8d4cef5e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:05:58.358178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0395e1e-fa8e-4931-987e-001a8d4cef5e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:58.359790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0395e1e-fa8e-4931-987e-001a8d4cef5e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:58.372236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 989/1991] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:05:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:05:58.382245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fe7ca0-3984-40b5-b8ab-256a0470c455 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:58.391455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fe7ca0-3984-40b5-b8ab-256a0470c455 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:58.396222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf495c8-8fff-4227-805d-12bbcd0a365a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:58.396898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf495c8-8fff-4227-805d-12bbcd0a365a admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:58.397184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf495c8-8fff-4227-805d-12bbcd0a365a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:58.397462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf495c8-8fff-4227-805d-12bbcd0a365a admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:58.398214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf495c8-8fff-4227-805d-12bbcd0a365a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:58.400179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf495c8-8fff-4227-805d-12bbcd0a365a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:58.487115 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1003/1992] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:05:58 2022] POST /identity/v3/projects => generated 383 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:58.491389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0395e1e-fa8e-4931-987e-001a8d4cef5e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:58.499139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:58.503111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0395e1e-fa8e-4931-987e-001a8d4cef5e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:58.559837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00002f3b-c395-4c04-9af7-b5930e993371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:58.560555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00002f3b-c395-4c04-9af7-b5930e993371 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:58.560870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00002f3b-c395-4c04-9af7-b5930e993371 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:58.561151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00002f3b-c395-4c04-9af7-b5930e993371 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:58.561837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00002f3b-c395-4c04-9af7-b5930e993371 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:58.563771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00002f3b-c395-4c04-9af7-b5930e993371 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:58.617163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 990/1993] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:05:58 2022] POST /identity/v3/users => generated 292 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:05:58.621224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf495c8-8fff-4227-805d-12bbcd0a365a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:58.629395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:58.633001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf495c8-8fff-4227-805d-12bbcd0a365a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:58.638220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633276ec-53b9-409f-a06b-6f4fa0c5c891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:58.638881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633276ec-53b9-409f-a06b-6f4fa0c5c891 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:58.639170 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633276ec-53b9-409f-a06b-6f4fa0c5c891 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:58.639448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633276ec-53b9-409f-a06b-6f4fa0c5c891 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:58.640180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633276ec-53b9-409f-a06b-6f4fa0c5c891 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:58.641862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-633276ec-53b9-409f-a06b-6f4fa0c5c891 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:58.663798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1004/1994] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:58 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:58.664104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00002f3b-c395-4c04-9af7-b5930e993371 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:58.667021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00002f3b-c395-4c04-9af7-b5930e993371 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:58.671644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-335ffeac-d96f-4302-a7ab-7a8f339ff0e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:58.672304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335ffeac-d96f-4302-a7ab-7a8f339ff0e6 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:58.672639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335ffeac-d96f-4302-a7ab-7a8f339ff0e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:58.672910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-335ffeac-d96f-4302-a7ab-7a8f339ff0e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:58.673667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335ffeac-d96f-4302-a7ab-7a8f339ff0e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:58.675300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-335ffeac-d96f-4302-a7ab-7a8f339ff0e6 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:58.685110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 991/1995] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:05:58 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:05:58.689365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633276ec-53b9-409f-a06b-6f4fa0c5c891 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:58.691485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:58.699740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-633276ec-53b9-409f-a06b-6f4fa0c5c891 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:58.704345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbe2a85-2ba4-43b9-8ca7-3ee740e3983f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:58.705249 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbe2a85-2ba4-43b9-8ca7-3ee740e3983f admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:58.705535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbe2a85-2ba4-43b9-8ca7-3ee740e3983f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:58.705816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fbe2a85-2ba4-43b9-8ca7-3ee740e3983f admin admin] PATH_INFO: `/v3/projects/388b72d80e3e44ce9aa3b5720fa7041a/users/c2350eadab924fa3a12291e2d2f9469d/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:58.721116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbe2a85-2ba4-43b9-8ca7-3ee740e3983f admin admin] RBAC: Authorizing `identity:create_grant(project_id=388b72d80e3e44ce9aa3b5720fa7041a, user_id=c2350eadab924fa3a12291e2d2f9469d, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:58.723797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fbe2a85-2ba4-43b9-8ca7-3ee740e3983f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:58.737746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:58.822833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1005/1996] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:05:58 2022] PUT /identity/v3/projects/388b72d80e3e44ce9aa3b5720fa7041a/users/c2350eadab924fa3a12291e2d2f9469d/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:05:58.837291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f97133-dcca-4109-b913-d8f1609d9031 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:58.837583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f97133-dcca-4109-b913-d8f1609d9031 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:58.837862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49f97133-dcca-4109-b913-d8f1609d9031 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:58.867018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-49f97133-dcca-4109-b913-d8f1609d9031 None None] MFA Rules not processed for user `c2350eadab924fa3a12291e2d2f9469d`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:05:58.868401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-49f97133-dcca-4109-b913-d8f1609d9031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:58.869056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:58.931273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 992/1997] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:05:58 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:05:58.942450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbe2a85-2ba4-43b9-8ca7-3ee740e3983f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:05:58.944715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5fbe2a85-2ba4-43b9-8ca7-3ee740e3983f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:05:58.982902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:59.000731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fbe2a85-2ba4-43b9-8ca7-3ee740e3983f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:05:59.004409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be62db72-27c4-41e0-a43d-41d67ba452e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:05:59.005098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be62db72-27c4-41e0-a43d-41d67ba452e1 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:05:59.005387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be62db72-27c4-41e0-a43d-41d67ba452e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:05:59.005668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be62db72-27c4-41e0-a43d-41d67ba452e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:05:59.014384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be62db72-27c4-41e0-a43d-41d67ba452e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:05:59.015828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be62db72-27c4-41e0-a43d-41d67ba452e1 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:05:59.024477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1006/1998] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:05:58 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:05:59.046257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:59.188905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:59.331709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:59.360356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:59.481991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:59.679174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:59.771832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:59.873184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:05:59.954408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.005746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:DELETE=7,keystone:SELECT=88,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.075337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.148566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.182479 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.251209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=91,keystone:INSERT=5,keystone:DELETE=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.274384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.382626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=84,keystone:INSERT=3,keystone:DELETE=13 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.399709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.468270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.497644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.565701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=1,keystone:DELETE=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.646120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.742790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=84,keystone:DELETE=13,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:00.893364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:01.188324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:01.239552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:01.284996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:08.812948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f97133-dcca-4109-b913-d8f1609d9031 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:08.823647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49f97133-dcca-4109-b913-d8f1609d9031 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:08.828676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f08f66-bc2d-4ab5-becb-17c6fbe1000f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:08.829378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f08f66-bc2d-4ab5-becb-17c6fbe1000f service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:08.829960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f08f66-bc2d-4ab5-becb-17c6fbe1000f service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:08.829960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f08f66-bc2d-4ab5-becb-17c6fbe1000f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:08.832606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-53f08f66-bc2d-4ab5-becb-17c6fbe1000f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:08.911220 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f08f66-bc2d-4ab5-becb-17c6fbe1000f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:08.914426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f08f66-bc2d-4ab5-becb-17c6fbe1000f service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:08.925826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 993/1999] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:08 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:11.068626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=35,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:11.778135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be62db72-27c4-41e0-a43d-41d67ba452e1 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:11.779552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-be62db72-27c4-41e0-a43d-41d67ba452e1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:11.790492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be62db72-27c4-41e0-a43d-41d67ba452e1 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:11.794193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6dc3d0-6aa7-4b77-85cc-2e143b8e57ef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:11.794844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6dc3d0-6aa7-4b77-85cc-2e143b8e57ef service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:11.795113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6dc3d0-6aa7-4b77-85cc-2e143b8e57ef service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:11.795389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6dc3d0-6aa7-4b77-85cc-2e143b8e57ef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:11.797114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-bc6dc3d0-6aa7-4b77-85cc-2e143b8e57ef service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:11.867004 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6dc3d0-6aa7-4b77-85cc-2e143b8e57ef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:11.868485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6dc3d0-6aa7-4b77-85cc-2e143b8e57ef service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:11.878372 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1007/2000] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:11 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:18.692387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f08f66-bc2d-4ab5-becb-17c6fbe1000f service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:18.759370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f08f66-bc2d-4ab5-becb-17c6fbe1000f service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:18.764420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:18.765326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:18.765615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:18.765895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] PATH_INFO: `/v3/users/c2350eadab924fa3a12291e2d2f9469d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:18.767085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2350eadab924fa3a12291e2d2f9469d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:18.769045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:18.967356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2350eadab924fa3a12291e2d2f9469d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:18.968017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2350eadab924fa3a12291e2d2f9469d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:18.968620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2350eadab924fa3a12291e2d2f9469d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:18.975697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2350eadab924fa3a12291e2d2f9469d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:18.977424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:19.096179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2350eadab924fa3a12291e2d2f9469d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:19.139803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 994/2001] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:18 2022] DELETE /identity/v3/users/c2350eadab924fa3a12291e2d2f9469d => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:19.143262 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ddb30e-8ae9-4d27-b724-b9bac3cf1d0c None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:19.143750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ddb30e-8ae9-4d27-b724-b9bac3cf1d0c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:19.143959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ddb30e-8ae9-4d27-b724-b9bac3cf1d0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:19.161763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-58ddb30e-8ae9-4d27-b724-b9bac3cf1d0c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:19.163058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-58ddb30e-8ae9-4d27-b724-b9bac3cf1d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:19.237483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1008/2002] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:19.246505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:19.248846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:19.307695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8391972-49e9-47c1-9a74-a1ea0755f296 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:19.312518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceff7a0f-29a1-41ec-936e-d26be7079100 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:19.313282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceff7a0f-29a1-41ec-936e-d26be7079100 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:19.313629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceff7a0f-29a1-41ec-936e-d26be7079100 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:19.313973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceff7a0f-29a1-41ec-936e-d26be7079100 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:19.323864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceff7a0f-29a1-41ec-936e-d26be7079100 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:19.325959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceff7a0f-29a1-41ec-936e-d26be7079100 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:19.339153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 995/2003] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:19.725921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ddb30e-8ae9-4d27-b724-b9bac3cf1d0c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:19.727451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-58ddb30e-8ae9-4d27-b724-b9bac3cf1d0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:19.745793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ddb30e-8ae9-4d27-b724-b9bac3cf1d0c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:19.762558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:19.763581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:19.763870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:19.764150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] PATH_INFO: `/v3/projects/388b72d80e3e44ce9aa3b5720fa7041a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:19.765393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] RBAC: Authorizing `identity:delete_project(project_id=388b72d80e3e44ce9aa3b5720fa7041a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:19.767709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:19.878533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '388b72d80e3e44ce9aa3b5720fa7041a', 'request_id': 'req-9f758bfc-5ce0-48bf-a62b-32fe416bff74'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:19.884717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '388b72d80e3e44ce9aa3b5720fa7041a', 'request_id': 'req-9f758bfc-5ce0-48bf-a62b-32fe416bff74'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:19.890147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '388b72d80e3e44ce9aa3b5720fa7041a', 'request_id': 'req-9f758bfc-5ce0-48bf-a62b-32fe416bff74'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:19.992302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] The token cache is being invalidate because project 388b72d80e3e44ce9aa3b5720fa7041a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:19.992592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:19.993259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9f758bfc-5ce0-48bf-a62b-32fe416bff74 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:19.994833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1009/2004] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:19 2022] DELETE /identity/v3/projects/388b72d80e3e44ce9aa3b5720fa7041a => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:20.000219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a43e14-d041-4ca3-bc3d-bad9017e36e9 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.000702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a43e14-d041-4ca3-bc3d-bad9017e36e9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.000830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a43e14-d041-4ca3-bc3d-bad9017e36e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.019802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-75a43e14-d041-4ca3-bc3d-bad9017e36e9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:20.021127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-75a43e14-d041-4ca3-bc3d-bad9017e36e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:20.080381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 996/2005] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:20.088331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe05e7c-49b8-4a42-a82a-4298417000b1 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.088625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe05e7c-49b8-4a42-a82a-4298417000b1 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.088944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe05e7c-49b8-4a42-a82a-4298417000b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.107786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-dbe05e7c-49b8-4a42-a82a-4298417000b1 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:20.109019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-dbe05e7c-49b8-4a42-a82a-4298417000b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:20.117121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1010/2006] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:20.121289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a43e14-d041-4ca3-bc3d-bad9017e36e9 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:20.133163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a43e14-d041-4ca3-bc3d-bad9017e36e9 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:20.138581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213f411a-0782-4841-80e5-0be9eeb71c23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:20.139251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213f411a-0782-4841-80e5-0be9eeb71c23 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.139532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213f411a-0782-4841-80e5-0be9eeb71c23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.139807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213f411a-0782-4841-80e5-0be9eeb71c23 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.140531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213f411a-0782-4841-80e5-0be9eeb71c23 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:20.140872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213f411a-0782-4841-80e5-0be9eeb71c23 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:20.142502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213f411a-0782-4841-80e5-0be9eeb71c23 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:20.155500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 997/2007] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:06:20.159848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe05e7c-49b8-4a42-a82a-4298417000b1 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:20.169687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe05e7c-49b8-4a42-a82a-4298417000b1 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:20.174261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0250208-2d01-45be-a2fa-409d4287cbc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:20.174957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0250208-2d01-45be-a2fa-409d4287cbc5 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.175237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0250208-2d01-45be-a2fa-409d4287cbc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.175763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0250208-2d01-45be-a2fa-409d4287cbc5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.183259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0250208-2d01-45be-a2fa-409d4287cbc5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:20.183259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0250208-2d01-45be-a2fa-409d4287cbc5 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:20.188172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:20.302906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1011/2008] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:20 2022] POST /identity/v3/projects => generated 373 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:20.306907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213f411a-0782-4841-80e5-0be9eeb71c23 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:20.317766 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213f411a-0782-4841-80e5-0be9eeb71c23 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:20.388910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5e2693-0297-4bed-ae0e-01926004adf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:20.389633 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5e2693-0297-4bed-ae0e-01926004adf5 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.389863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5e2693-0297-4bed-ae0e-01926004adf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.390147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5e2693-0297-4bed-ae0e-01926004adf5 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.390843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5e2693-0297-4bed-ae0e-01926004adf5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:20.392787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5e2693-0297-4bed-ae0e-01926004adf5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:20.403676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=94 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:20.484453 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 998/2009] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:20 2022] POST /identity/v3/users => generated 287 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:20.488714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0250208-2d01-45be-a2fa-409d4287cbc5 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:20.499627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0250208-2d01-45be-a2fa-409d4287cbc5 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:20.504298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4435781e-2fe8-476b-beeb-9e79a07d871d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:20.504928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4435781e-2fe8-476b-beeb-9e79a07d871d admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.505236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4435781e-2fe8-476b-beeb-9e79a07d871d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.505498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4435781e-2fe8-476b-beeb-9e79a07d871d admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.506269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4435781e-2fe8-476b-beeb-9e79a07d871d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:20.507862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4435781e-2fe8-476b-beeb-9e79a07d871d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:20.518885 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1012/2010] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:20.522648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5e2693-0297-4bed-ae0e-01926004adf5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:20.532582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5e2693-0297-4bed-ae0e-01926004adf5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:20.537206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c60f7332-087e-4f9b-8584-dc78a4f49a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:20.537853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60f7332-087e-4f9b-8584-dc78a4f49a85 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.538139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60f7332-087e-4f9b-8584-dc78a4f49a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.538416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c60f7332-087e-4f9b-8584-dc78a4f49a85 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.539161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60f7332-087e-4f9b-8584-dc78a4f49a85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:20.540858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c60f7332-087e-4f9b-8584-dc78a4f49a85 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:20.551241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 999/2011] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:20.555382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4435781e-2fe8-476b-beeb-9e79a07d871d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:20.565847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4435781e-2fe8-476b-beeb-9e79a07d871d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:20.570478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1659699-bfa7-47e3-ba3c-75e7076bd567 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:20.571348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1659699-bfa7-47e3-ba3c-75e7076bd567 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.571636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1659699-bfa7-47e3-ba3c-75e7076bd567 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.571916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1659699-bfa7-47e3-ba3c-75e7076bd567 admin admin] PATH_INFO: `/v3/projects/d1658a9d6de94030abaf2e7f5c82fc75/users/07363a81c7e844468ae3e2c203711d78/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.586890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1659699-bfa7-47e3-ba3c-75e7076bd567 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d1658a9d6de94030abaf2e7f5c82fc75, user_id=07363a81c7e844468ae3e2c203711d78, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:20.589587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1659699-bfa7-47e3-ba3c-75e7076bd567 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:20.644352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1013/2012] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:20 2022] PUT /identity/v3/projects/d1658a9d6de94030abaf2e7f5c82fc75/users/07363a81c7e844468ae3e2c203711d78/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:20.654858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b216c5fd-2f5e-4c58-921b-22b222675d9f None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.655151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b216c5fd-2f5e-4c58-921b-22b222675d9f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.655436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b216c5fd-2f5e-4c58-921b-22b222675d9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.681767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-b216c5fd-2f5e-4c58-921b-22b222675d9f None None] MFA Rules not processed for user `07363a81c7e844468ae3e2c203711d78`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:20.682971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b216c5fd-2f5e-4c58-921b-22b222675d9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:20.745292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1000/2013] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:20 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:20.755856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1659699-bfa7-47e3-ba3c-75e7076bd567 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:20.757502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f1659699-bfa7-47e3-ba3c-75e7076bd567 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:20.817252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1659699-bfa7-47e3-ba3c-75e7076bd567 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:20.820886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408dad33-3f42-4ec5-ac93-29d76d4ab57c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:20.821526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408dad33-3f42-4ec5-ac93-29d76d4ab57c service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:20.821809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408dad33-3f42-4ec5-ac93-29d76d4ab57c service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:20.822085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408dad33-3f42-4ec5-ac93-29d76d4ab57c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:20.830426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408dad33-3f42-4ec5-ac93-29d76d4ab57c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:20.831841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408dad33-3f42-4ec5-ac93-29d76d4ab57c service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:20.841109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1014/2014] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:20 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:23.694490 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c2b02c-6da0-4168-9500-31aabfb07235 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:23.694956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c2b02c-6da0-4168-9500-31aabfb07235 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:23.695200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c2b02c-6da0-4168-9500-31aabfb07235 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:23.715775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-d7c2b02c-6da0-4168-9500-31aabfb07235 None None] MFA Rules not processed for user `07363a81c7e844468ae3e2c203711d78`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:23.717077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d7c2b02c-6da0-4168-9500-31aabfb07235 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:23.719018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408dad33-3f42-4ec5-ac93-29d76d4ab57c service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:23.720561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-408dad33-3f42-4ec5-ac93-29d76d4ab57c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:23.730397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1001/2015] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:23 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:23.796113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408dad33-3f42-4ec5-ac93-29d76d4ab57c service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:23.800708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:23.801636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:23.801921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:23.802198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] PATH_INFO: `/v3/users/624700c503f5495dbb215d23557c6968` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:23.803571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] RBAC: Authorizing `identity:delete_user(user_id=624700c503f5495dbb215d23557c6968)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:23.805640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:23.972210 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '624700c503f5495dbb215d23557c6968'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:23.972858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '624700c503f5495dbb215d23557c6968'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:23.988711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '624700c503f5495dbb215d23557c6968'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:23.989028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:24.039293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '624700c503f5495dbb215d23557c6968'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:24.040348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '624700c503f5495dbb215d23557c6968'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:24.059977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1015/2016] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:23 2022] DELETE /identity/v3/users/624700c503f5495dbb215d23557c6968 => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:06:24.064387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326c8041-1a9f-4576-85cd-61c5eb51e8d5 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:24.064683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326c8041-1a9f-4576-85cd-61c5eb51e8d5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:24.065003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326c8041-1a9f-4576-85cd-61c5eb51e8d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:24.082347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-326c8041-1a9f-4576-85cd-61c5eb51e8d5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:24.083665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-326c8041-1a9f-4576-85cd-61c5eb51e8d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:24.146489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1002/2017] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:24.156440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:24.158027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:24.216276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc18dae-eca7-4b92-9f96-ec39d4adab47 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:24.220026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f8df49-7539-4e6e-9097-4bf9abdaf6c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:24.220700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f8df49-7539-4e6e-9097-4bf9abdaf6c4 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:24.221034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f8df49-7539-4e6e-9097-4bf9abdaf6c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:24.221334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f8df49-7539-4e6e-9097-4bf9abdaf6c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:24.231395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f8df49-7539-4e6e-9097-4bf9abdaf6c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:24.232821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68f8df49-7539-4e6e-9097-4bf9abdaf6c4 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:24.242491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1016/2018] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:24 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:06:24.697062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326c8041-1a9f-4576-85cd-61c5eb51e8d5 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:24.699391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-326c8041-1a9f-4576-85cd-61c5eb51e8d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:24.721248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326c8041-1a9f-4576-85cd-61c5eb51e8d5 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:24.746062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:24.747578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:24.747869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:24.748152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] PATH_INFO: `/v3/projects/c542fc3d32b8464a944116d6beffc8bf` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:24.749399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c542fc3d32b8464a944116d6beffc8bf)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:24.751396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:24.852661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c542fc3d32b8464a944116d6beffc8bf', 'request_id': 'req-10e876ec-90e7-46aa-be7c-418066267a33'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:24.950631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f8df49-7539-4e6e-9097-4bf9abdaf6c4 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:24.952244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-68f8df49-7539-4e6e-9097-4bf9abdaf6c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:24.958927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c542fc3d32b8464a944116d6beffc8bf', 'request_id': 'req-10e876ec-90e7-46aa-be7c-418066267a33'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:24.967022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c542fc3d32b8464a944116d6beffc8bf', 'request_id': 'req-10e876ec-90e7-46aa-be7c-418066267a33'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:24.968112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68f8df49-7539-4e6e-9097-4bf9abdaf6c4 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:24.968895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:INSERT=2,keystone:SELECT=93,keystone:DELETE=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:24.984518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] The token cache is being invalidate because project c542fc3d32b8464a944116d6beffc8bf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:24.984720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:24.985602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:24.988194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1003/2019] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:24 2022] DELETE /identity/v3/projects/c542fc3d32b8464a944116d6beffc8bf => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:24.991894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:24.995889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:25.042605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:25.044319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:25.045404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:25.046205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] PATH_INFO: `/v3/users/625f54352dcf4ae5a0bc3641b56c14ae` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:25.053104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=625f54352dcf4ae5a0bc3641b56c14ae)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:25.056064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:25.077358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e876ec-90e7-46aa-be7c-418066267a33 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:25.083233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:25.084111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:25.084650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:25.085037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] PATH_INFO: `/v3/users/d502709541734ea0afb87adec4dbe771` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:25.086292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d502709541734ea0afb87adec4dbe771)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:25.088289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:25.352945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '625f54352dcf4ae5a0bc3641b56c14ae'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.353815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '625f54352dcf4ae5a0bc3641b56c14ae'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.355728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd502709541734ea0afb87adec4dbe771'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.356988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd502709541734ea0afb87adec4dbe771'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.357711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd502709541734ea0afb87adec4dbe771'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.361892 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '625f54352dcf4ae5a0bc3641b56c14ae'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.364844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd502709541734ea0afb87adec4dbe771'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.427486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '625f54352dcf4ae5a0bc3641b56c14ae'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.428628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '625f54352dcf4ae5a0bc3641b56c14ae'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.450286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:25.455901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1017/2020] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:24 2022] DELETE /identity/v3/users/625f54352dcf4ae5a0bc3641b56c14ae => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:06:25.456028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:25.457541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:25.514713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b6b61f60-d597-48b6-b5d1-567f34839ff6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd502709541734ea0afb87adec4dbe771'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.526990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8419d312-359b-46d1-ac96-a0e20f63b1f3 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:25.533854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:25.534670 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:25.534963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:25.539697 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] PATH_INFO: `/v3/users/43c31865d2d744d6bb809a7048d2c575` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:25.539804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1004/2021] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:24 2022] DELETE /identity/v3/users/d502709541734ea0afb87adec4dbe771 => generated 0 bytes in 545 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:06:25.539929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=43c31865d2d744d6bb809a7048d2c575)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:25.540105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3265e786-cd5f-4b9d-af9f-8d91ff17ba61 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:25.540105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3265e786-cd5f-4b9d-af9f-8d91ff17ba61 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:25.540105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3265e786-cd5f-4b9d-af9f-8d91ff17ba61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:25.540299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:25.558295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3265e786-cd5f-4b9d-af9f-8d91ff17ba61 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:25.559560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3265e786-cd5f-4b9d-af9f-8d91ff17ba61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:25.623711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1005/2022] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:25.633268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3265e786-cd5f-4b9d-af9f-8d91ff17ba61 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:25.634697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3265e786-cd5f-4b9d-af9f-8d91ff17ba61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:25.645293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:25.690620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3265e786-cd5f-4b9d-af9f-8d91ff17ba61 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:25.694298 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7759be-f7d5-439d-ae95-c435b4f022a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:25.694884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7759be-f7d5-439d-ae95-c435b4f022a4 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:25.695168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7759be-f7d5-439d-ae95-c435b4f022a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:25.695555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7759be-f7d5-439d-ae95-c435b4f022a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:25.705377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7759be-f7d5-439d-ae95-c435b4f022a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:25.706817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7759be-f7d5-439d-ae95-c435b4f022a4 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:25.717019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1006/2023] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:06:25.729234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '43c31865d2d744d6bb809a7048d2c575'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.729933 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '43c31865d2d744d6bb809a7048d2c575'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.741259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '43c31865d2d744d6bb809a7048d2c575'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.799097 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '43c31865d2d744d6bb809a7048d2c575'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.801102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '43c31865d2d744d6bb809a7048d2c575'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:25.823198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1018/2024] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:25 2022] DELETE /identity/v3/users/43c31865d2d744d6bb809a7048d2c575 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:26.011048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7759be-f7d5-439d-ae95-c435b4f022a4 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:26.013283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0a7759be-f7d5-439d-ae95-c435b4f022a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:26.026151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7759be-f7d5-439d-ae95-c435b4f022a4 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:26.104637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:26.105554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:26.105859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:26.106135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] PATH_INFO: `/v3/projects/7f2b65d4a7ab4359983515c792bf42d3` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:26.107335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f2b65d4a7ab4359983515c792bf42d3)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:26.109480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:26.118839 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:26.120749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:26.136047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e753197d-19e1-45ef-951a-77305adbf5c0 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:26.153728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:26.153728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:26.154316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:26.154799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] PATH_INFO: `/v3/projects/72affe24c05f4972a9a30883d2cbc85f` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:26.156605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] RBAC: Authorizing `identity:delete_project(project_id=72affe24c05f4972a9a30883d2cbc85f)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:26.158606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:26.250840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f2b65d4a7ab4359983515c792bf42d3', 'request_id': 'req-47170a3d-ac23-480a-bfa4-c663e41a6303'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.273548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72affe24c05f4972a9a30883d2cbc85f', 'request_id': 'req-ea14e40e-e3f4-4999-8637-4058a63b5f47'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.279809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72affe24c05f4972a9a30883d2cbc85f', 'request_id': 'req-ea14e40e-e3f4-4999-8637-4058a63b5f47'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.285582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72affe24c05f4972a9a30883d2cbc85f', 'request_id': 'req-ea14e40e-e3f4-4999-8637-4058a63b5f47'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.324623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f2b65d4a7ab4359983515c792bf42d3', 'request_id': 'req-47170a3d-ac23-480a-bfa4-c663e41a6303'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.332155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f2b65d4a7ab4359983515c792bf42d3', 'request_id': 'req-47170a3d-ac23-480a-bfa4-c663e41a6303'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.342868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] The token cache is being invalidate because project 7f2b65d4a7ab4359983515c792bf42d3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:26.343159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.343837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.345413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1007/2025] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:26 2022] DELETE /identity/v3/projects/7f2b65d4a7ab4359983515c792bf42d3 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:26.346421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:26.347904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:26.383258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:26.387431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] The token cache is being invalidate because project 72affe24c05f4972a9a30883d2cbc85f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:26.387861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.388582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.390209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1019/2026] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:26 2022] DELETE /identity/v3/projects/72affe24c05f4972a9a30883d2cbc85f => generated 0 bytes in 272 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:06:26.390978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:26.392371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:26.394448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:26.413012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47170a3d-ac23-480a-bfa4-c663e41a6303 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:26.459355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea14e40e-e3f4-4999-8637-4058a63b5f47 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:26.464390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:26.465340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:26.465602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:26.465897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] PATH_INFO: `/v3/users/fb3ee230dcec4ff79d073321f003491e` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:26.467099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] RBAC: Authorizing `identity:delete_user(user_id=fb3ee230dcec4ff79d073321f003491e)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:26.469062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:26.475388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:26.476251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:26.476532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:26.476835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] PATH_INFO: `/v3/projects/48c25574ef6c447b9b0aa4dd366fa7af` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:26.478149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=48c25574ef6c447b9b0aa4dd366fa7af)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:26.480422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:26.686102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '48c25574ef6c447b9b0aa4dd366fa7af', 'request_id': 'req-1bf55bd2-5981-4577-b870-3a0ed988eb14'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.686852 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fb3ee230dcec4ff79d073321f003491e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.687516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fb3ee230dcec4ff79d073321f003491e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.694789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fb3ee230dcec4ff79d073321f003491e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.770235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '48c25574ef6c447b9b0aa4dd366fa7af', 'request_id': 'req-1bf55bd2-5981-4577-b870-3a0ed988eb14'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.777174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '48c25574ef6c447b9b0aa4dd366fa7af', 'request_id': 'req-1bf55bd2-5981-4577-b870-3a0ed988eb14'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.788342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] The token cache is being invalidate because project 48c25574ef6c447b9b0aa4dd366fa7af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:26.788571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.789265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf55bd2-5981-4577-b870-3a0ed988eb14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.790869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1008/2027] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:26 2022] DELETE /identity/v3/projects/48c25574ef6c447b9b0aa4dd366fa7af => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:26.794301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861cea38-6dff-4f55-989e-160c4ccd6323 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:26.794301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861cea38-6dff-4f55-989e-160c4ccd6323 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:26.794301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861cea38-6dff-4f55-989e-160c4ccd6323 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:26.810924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-861cea38-6dff-4f55-989e-160c4ccd6323 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:26.812135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-861cea38-6dff-4f55-989e-160c4ccd6323 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:26.827237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fb3ee230dcec4ff79d073321f003491e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.830043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fb3ee230dcec4ff79d073321f003491e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:26.847083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1020/2028] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:26 2022] DELETE /identity/v3/users/fb3ee230dcec4ff79d073321f003491e => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:26.847830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:26.849357 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:26.873698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1009/2029] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:26.881479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efae658d-e738-4c33-b289-4b97629c0ef1 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:26.881660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efae658d-e738-4c33-b289-4b97629c0ef1 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:26.881947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efae658d-e738-4c33-b289-4b97629c0ef1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:26.907752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-efae658d-e738-4c33-b289-4b97629c0ef1 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:26.910189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-efae658d-e738-4c33-b289-4b97629c0ef1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:26.929218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:26.940538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d984b3-5ed8-4a59-9e03-bdc87368481c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:26.948791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:26.950880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:26.951173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:26.951453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] PATH_INFO: `/v3/users/6d610ef444c14efaa43dbd2ed0b667f6` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:26.952702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6d610ef444c14efaa43dbd2ed0b667f6)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:26.954751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:26.976998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:26.979465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1010/2030] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:26.983674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efae658d-e738-4c33-b289-4b97629c0ef1 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:26.992833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efae658d-e738-4c33-b289-4b97629c0ef1 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:26.997660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5df4fce-0540-4938-93d2-b4c451d68bd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:26.998295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df4fce-0540-4938-93d2-b4c451d68bd8 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:26.998576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df4fce-0540-4938-93d2-b4c451d68bd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:26.998849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df4fce-0540-4938-93d2-b4c451d68bd8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:26.999638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5df4fce-0540-4938-93d2-b4c451d68bd8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:26.999939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5df4fce-0540-4938-93d2-b4c451d68bd8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.001599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5df4fce-0540-4938-93d2-b4c451d68bd8 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.013759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1011/2031] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:06:27.017748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5df4fce-0540-4938-93d2-b4c451d68bd8 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.026866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5df4fce-0540-4938-93d2-b4c451d68bd8 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.032105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee29e0f8-bccf-45fe-a94c-439340362ff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.032743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee29e0f8-bccf-45fe-a94c-439340362ff0 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.033070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee29e0f8-bccf-45fe-a94c-439340362ff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.033348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee29e0f8-bccf-45fe-a94c-439340362ff0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.034019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee29e0f8-bccf-45fe-a94c-439340362ff0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.036819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee29e0f8-bccf-45fe-a94c-439340362ff0 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.177377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1012/2032] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:27 2022] POST /identity/v3/projects => generated 391 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:27.181450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee29e0f8-bccf-45fe-a94c-439340362ff0 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.188332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6d610ef444c14efaa43dbd2ed0b667f6'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.189444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6d610ef444c14efaa43dbd2ed0b667f6'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.196674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6d610ef444c14efaa43dbd2ed0b667f6'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.250657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee29e0f8-bccf-45fe-a94c-439340362ff0 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.255230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7e8a8f-3d89-4151-9a02-cc48118b54a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.255895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7e8a8f-3d89-4151-9a02-cc48118b54a1 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.256178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7e8a8f-3d89-4151-9a02-cc48118b54a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.256454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7e8a8f-3d89-4151-9a02-cc48118b54a1 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.257177 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7e8a8f-3d89-4151-9a02-cc48118b54a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.259131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7e8a8f-3d89-4151-9a02-cc48118b54a1 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.308816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6d610ef444c14efaa43dbd2ed0b667f6'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.309451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6d610ef444c14efaa43dbd2ed0b667f6'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.327363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1021/2033] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:26 2022] DELETE /identity/v3/users/6d610ef444c14efaa43dbd2ed0b667f6 => generated 0 bytes in 480 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:27.328116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.329494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:27.382191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1013/2034] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:27 2022] POST /identity/v3/users => generated 296 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:27.386252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7e8a8f-3d89-4151-9a02-cc48118b54a1 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.388159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2b7e8a8f-3d89-4151-9a02-cc48118b54a1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:27.399682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97b210ba-b067-4f23-8fbd-8e7e34da4ca7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.400902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7e8a8f-3d89-4151-9a02-cc48118b54a1 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.405323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.406193 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.406486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.406798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] PATH_INFO: `/v3/projects/7012d2f7c89a4bf2afc4e500980d6223` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.408008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7012d2f7c89a4bf2afc4e500980d6223)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.408127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcb1769-c094-4891-a50e-8f50af467f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.408704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcb1769-c094-4891-a50e-8f50af467f3e admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.409058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcb1769-c094-4891-a50e-8f50af467f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.409346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcb1769-c094-4891-a50e-8f50af467f3e admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.410086 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcb1769-c094-4891-a50e-8f50af467f3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.410197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.411700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcb1769-c094-4891-a50e-8f50af467f3e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.428242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1014/2035] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:27 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:06:27.432019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcb1769-c094-4891-a50e-8f50af467f3e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.443331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcb1769-c094-4891-a50e-8f50af467f3e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.448158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b685d7ee-4b9b-4c56-8f65-6b2d8939e7d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.448850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685d7ee-4b9b-4c56-8f65-6b2d8939e7d0 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.449424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685d7ee-4b9b-4c56-8f65-6b2d8939e7d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.449623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b685d7ee-4b9b-4c56-8f65-6b2d8939e7d0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.450311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b685d7ee-4b9b-4c56-8f65-6b2d8939e7d0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.451962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b685d7ee-4b9b-4c56-8f65-6b2d8939e7d0 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.462886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1015/2036] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:06:27.467448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b685d7ee-4b9b-4c56-8f65-6b2d8939e7d0 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.477171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b685d7ee-4b9b-4c56-8f65-6b2d8939e7d0 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.482518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1cd98f2-0e6d-4ab5-bf66-927d066f827d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.483382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cd98f2-0e6d-4ab5-bf66-927d066f827d admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.483669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cd98f2-0e6d-4ab5-bf66-927d066f827d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.483948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1cd98f2-0e6d-4ab5-bf66-927d066f827d admin admin] PATH_INFO: `/v3/projects/03423e3482614374afa8c929bd931262/users/25f73d0bd4e64057a922d11be46eb04c/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.498352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cd98f2-0e6d-4ab5-bf66-927d066f827d admin admin] RBAC: Authorizing `identity:create_grant(project_id=03423e3482614374afa8c929bd931262, user_id=25f73d0bd4e64057a922d11be46eb04c, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.501151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1cd98f2-0e6d-4ab5-bf66-927d066f827d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.609580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1016/2037] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:27 2022] PUT /identity/v3/projects/03423e3482614374afa8c929bd931262/users/25f73d0bd4e64057a922d11be46eb04c/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:06:27.615372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e77aeb0-4e04-478d-9a26-c74ce01a163a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.615791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e77aeb0-4e04-478d-9a26-c74ce01a163a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.616402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e77aeb0-4e04-478d-9a26-c74ce01a163a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.620242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7012d2f7c89a4bf2afc4e500980d6223', 'request_id': 'req-1317c78f-1b77-4442-9816-b844ec110937'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.627983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7012d2f7c89a4bf2afc4e500980d6223', 'request_id': 'req-1317c78f-1b77-4442-9816-b844ec110937'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.636360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7012d2f7c89a4bf2afc4e500980d6223', 'request_id': 'req-1317c78f-1b77-4442-9816-b844ec110937'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.641342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5e77aeb0-4e04-478d-9a26-c74ce01a163a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:27.645054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5e77aeb0-4e04-478d-9a26-c74ce01a163a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:27.669975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:27.712804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1017/2038] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:27.725479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e77aeb0-4e04-478d-9a26-c74ce01a163a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.727113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5e77aeb0-4e04-478d-9a26-c74ce01a163a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:27.740283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] The token cache is being invalidate because project 7012d2f7c89a4bf2afc4e500980d6223 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:27.740578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.741293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1317c78f-1b77-4442-9816-b844ec110937 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:27.745308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1022/2039] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:27 2022] DELETE /identity/v3/projects/7012d2f7c89a4bf2afc4e500980d6223 => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:27.750085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f5fb95-1f09-4416-9466-d3402581fd79 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.750373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f5fb95-1f09-4416-9466-d3402581fd79 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.750860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f5fb95-1f09-4416-9466-d3402581fd79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.768299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-b1f5fb95-1f09-4416-9466-d3402581fd79 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:27.769638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b1f5fb95-1f09-4416-9466-d3402581fd79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:27.779359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1023/2040] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:27.791821 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e77aeb0-4e04-478d-9a26-c74ce01a163a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.795236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c6c6d4-c655-469c-b31f-4edb213515fc None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.795525 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c6c6d4-c655-469c-b31f-4edb213515fc None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.795800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c6c6d4-c655-469c-b31f-4edb213515fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.798991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c8bbe8-4749-4d5b-b11c-5678a1e2c2a8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.799822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c8bbe8-4749-4d5b-b11c-5678a1e2c2a8 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.800334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c8bbe8-4749-4d5b-b11c-5678a1e2c2a8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.800715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c8bbe8-4749-4d5b-b11c-5678a1e2c2a8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.802837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-44c8bbe8-4749-4d5b-b11c-5678a1e2c2a8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:27.816831 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c8bbe8-4749-4d5b-b11c-5678a1e2c2a8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.817039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-13c6c6d4-c655-469c-b31f-4edb213515fc None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:27.818626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-13c6c6d4-c655-469c-b31f-4edb213515fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:27.818943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44c8bbe8-4749-4d5b-b11c-5678a1e2c2a8 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.827617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1024/2041] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:27.829687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1018/2042] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:27.831473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c6c6d4-c655-469c-b31f-4edb213515fc None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.831473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-13c6c6d4-c655-469c-b31f-4edb213515fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:27.834268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c8bbe8-4749-4d5b-b11c-5678a1e2c2a8 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.843443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c6c6d4-c655-469c-b31f-4edb213515fc None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.844202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c8bbe8-4749-4d5b-b11c-5678a1e2c2a8 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.848113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.849125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.849288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404fad97-d010-4e00-806f-78296657d01d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.849403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.849662 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] PATH_INFO: `/v3/projects/71fbd80ef4fa46fdb8dccec1d7889132` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.849958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404fad97-d010-4e00-806f-78296657d01d admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.850237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404fad97-d010-4e00-806f-78296657d01d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.850647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-404fad97-d010-4e00-806f-78296657d01d admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.850865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=71fbd80ef4fa46fdb8dccec1d7889132)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.851554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404fad97-d010-4e00-806f-78296657d01d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:27.851865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404fad97-d010-4e00-806f-78296657d01d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.852801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.853509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-404fad97-d010-4e00-806f-78296657d01d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:27.864854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:27.872825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1019/2043] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:06:27.873578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404fad97-d010-4e00-806f-78296657d01d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:27.883282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-404fad97-d010-4e00-806f-78296657d01d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:27.887959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9464d3ad-9a3c-4589-a398-05329c281508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:27.888599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9464d3ad-9a3c-4589-a398-05329c281508 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:27.888926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9464d3ad-9a3c-4589-a398-05329c281508 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:27.889208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9464d3ad-9a3c-4589-a398-05329c281508 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:27.889892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9464d3ad-9a3c-4589-a398-05329c281508 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:27.891808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9464d3ad-9a3c-4589-a398-05329c281508 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.036735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1020/2044] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:27 2022] POST /identity/v3/projects => generated 369 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:28.047670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9464d3ad-9a3c-4589-a398-05329c281508 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.049081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '71fbd80ef4fa46fdb8dccec1d7889132', 'request_id': 'req-21c06468-d746-437f-996e-0c77dcc32de9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:28.056081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '71fbd80ef4fa46fdb8dccec1d7889132', 'request_id': 'req-21c06468-d746-437f-996e-0c77dcc32de9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:28.061924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '71fbd80ef4fa46fdb8dccec1d7889132', 'request_id': 'req-21c06468-d746-437f-996e-0c77dcc32de9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:28.103395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9464d3ad-9a3c-4589-a398-05329c281508 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.108785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d3b85e-e744-4032-af19-5b2501be861f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.109444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d3b85e-e744-4032-af19-5b2501be861f admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.109727 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d3b85e-e744-4032-af19-5b2501be861f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.110003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d3b85e-e744-4032-af19-5b2501be861f admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.110679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d3b85e-e744-4032-af19-5b2501be861f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.113069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d3b85e-e744-4032-af19-5b2501be861f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.176518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] The token cache is being invalidate because project 71fbd80ef4fa46fdb8dccec1d7889132 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:28.176935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:28.177646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-21c06468-d746-437f-996e-0c77dcc32de9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:28.179473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1025/2045] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:27 2022] DELETE /identity/v3/projects/71fbd80ef4fa46fdb8dccec1d7889132 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:28.184971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93454c02-6803-4e7c-bdd2-0875a8895f4a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.185382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93454c02-6803-4e7c-bdd2-0875a8895f4a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.185981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93454c02-6803-4e7c-bdd2-0875a8895f4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.205779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-93454c02-6803-4e7c-bdd2-0875a8895f4a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:28.209350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-93454c02-6803-4e7c-bdd2-0875a8895f4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:28.220911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1026/2046] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:28.227131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf20c5d-e0d1-4eea-a9a4-a300c6e26688 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.227487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf20c5d-e0d1-4eea-a9a4-a300c6e26688 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.227837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abf20c5d-e0d1-4eea-a9a4-a300c6e26688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.247291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-abf20c5d-e0d1-4eea-a9a4-a300c6e26688 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:28.248604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-abf20c5d-e0d1-4eea-a9a4-a300c6e26688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:28.256882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1027/2047] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:28.261066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf20c5d-e0d1-4eea-a9a4-a300c6e26688 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.270425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abf20c5d-e0d1-4eea-a9a4-a300c6e26688 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.275582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c3051d-4721-42e3-9a50-ccdbf3e039e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.278396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c3051d-4721-42e3-9a50-ccdbf3e039e7 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.278396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c3051d-4721-42e3-9a50-ccdbf3e039e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.278396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c3051d-4721-42e3-9a50-ccdbf3e039e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.278396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c3051d-4721-42e3-9a50-ccdbf3e039e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:28.278813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c3051d-4721-42e3-9a50-ccdbf3e039e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.281047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c3051d-4721-42e3-9a50-ccdbf3e039e7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.294125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1028/2048] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:06:28.298858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c3051d-4721-42e3-9a50-ccdbf3e039e7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.310691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c3051d-4721-42e3-9a50-ccdbf3e039e7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.315469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017ff3c6-4cef-4527-9a5a-84431c376662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.316144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017ff3c6-4cef-4527-9a5a-84431c376662 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.316428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017ff3c6-4cef-4527-9a5a-84431c376662 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.316704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-017ff3c6-4cef-4527-9a5a-84431c376662 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.317429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017ff3c6-4cef-4527-9a5a-84431c376662 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.319392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-017ff3c6-4cef-4527-9a5a-84431c376662 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.453061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1029/2049] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:28 2022] POST /identity/v3/projects => generated 367 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:28.456740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017ff3c6-4cef-4527-9a5a-84431c376662 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.484567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1021/2050] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:28 2022] POST /identity/v3/users => generated 285 bytes in 445 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:28.488968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d3b85e-e744-4032-af19-5b2501be861f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.490905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-61d3b85e-e744-4032-af19-5b2501be861f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:28.510807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d3b85e-e744-4032-af19-5b2501be861f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.536148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-017ff3c6-4cef-4527-9a5a-84431c376662 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.553560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923405a9-4682-48d1-a49e-a3520996d2d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.554259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923405a9-4682-48d1-a49e-a3520996d2d7 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.554547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923405a9-4682-48d1-a49e-a3520996d2d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.554831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-923405a9-4682-48d1-a49e-a3520996d2d7 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.555548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923405a9-4682-48d1-a49e-a3520996d2d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.557515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-923405a9-4682-48d1-a49e-a3520996d2d7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.587793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16037386-0c68-43dc-a5f3-265bc996e208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.588580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16037386-0c68-43dc-a5f3-265bc996e208 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.588819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16037386-0c68-43dc-a5f3-265bc996e208 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.589106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16037386-0c68-43dc-a5f3-265bc996e208 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.589885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16037386-0c68-43dc-a5f3-265bc996e208 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.592167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16037386-0c68-43dc-a5f3-265bc996e208 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.605200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1022/2051] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:28 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:28.606591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16037386-0c68-43dc-a5f3-265bc996e208 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.616747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16037386-0c68-43dc-a5f3-265bc996e208 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.621622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe20c49-cce6-42fb-85df-515361f8e243 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.622444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe20c49-cce6-42fb-85df-515361f8e243 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.622740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe20c49-cce6-42fb-85df-515361f8e243 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.624192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:28.624756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe20c49-cce6-42fb-85df-515361f8e243 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.625715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe20c49-cce6-42fb-85df-515361f8e243 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.627530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fe20c49-cce6-42fb-85df-515361f8e243 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.638766 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1023/2052] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:28 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:28.642630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe20c49-cce6-42fb-85df-515361f8e243 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.652105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe20c49-cce6-42fb-85df-515361f8e243 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.657562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11beb6f-1950-44c5-9835-f876b7422ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.658513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11beb6f-1950-44c5-9835-f876b7422ef2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.658792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11beb6f-1950-44c5-9835-f876b7422ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.659958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11beb6f-1950-44c5-9835-f876b7422ef2 admin admin] PATH_INFO: `/v3/projects/95e50863a8ad44ce84e13b85301e0dc9/users/74dd023f4a024e8898e1d4b55cc71d60/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.673902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11beb6f-1950-44c5-9835-f876b7422ef2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95e50863a8ad44ce84e13b85301e0dc9, user_id=74dd023f4a024e8898e1d4b55cc71d60, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.676654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11beb6f-1950-44c5-9835-f876b7422ef2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.685046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1030/2053] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:28 2022] POST /identity/v3/users => generated 284 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:28.688966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923405a9-4682-48d1-a49e-a3520996d2d7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.698349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-923405a9-4682-48d1-a49e-a3520996d2d7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.703440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fde8cca-dd19-41f5-88d8-45e36e346b79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.704081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fde8cca-dd19-41f5-88d8-45e36e346b79 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.704363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fde8cca-dd19-41f5-88d8-45e36e346b79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.711690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fde8cca-dd19-41f5-88d8-45e36e346b79 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.711690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fde8cca-dd19-41f5-88d8-45e36e346b79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.711690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fde8cca-dd19-41f5-88d8-45e36e346b79 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.717985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1031/2054] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:06:28.721492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fde8cca-dd19-41f5-88d8-45e36e346b79 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.730974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fde8cca-dd19-41f5-88d8-45e36e346b79 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.747734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c31b2-9413-4f16-a9d4-976ec791f2eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.748424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c31b2-9413-4f16-a9d4-976ec791f2eb admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.748713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c31b2-9413-4f16-a9d4-976ec791f2eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.749019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874c31b2-9413-4f16-a9d4-976ec791f2eb admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.749756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874c31b2-9413-4f16-a9d4-976ec791f2eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.751484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874c31b2-9413-4f16-a9d4-976ec791f2eb admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.765124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1032/2055] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:28 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:06:28.769054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c31b2-9413-4f16-a9d4-976ec791f2eb admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.773659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1024/2056] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:28 2022] PUT /identity/v3/projects/95e50863a8ad44ce84e13b85301e0dc9/users/74dd023f4a024e8898e1d4b55cc71d60/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:28.782925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874c31b2-9413-4f16-a9d4-976ec791f2eb admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:28.783806 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3520788-ab83-46f7-abfb-3696e2fff506 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.784102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3520788-ab83-46f7-abfb-3696e2fff506 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.784414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3520788-ab83-46f7-abfb-3696e2fff506 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.813546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a3520788-ab83-46f7-abfb-3696e2fff506 None None] MFA Rules not processed for user `74dd023f4a024e8898e1d4b55cc71d60`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:28.814814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a3520788-ab83-46f7-abfb-3696e2fff506 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:28.847991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3255c162-dacd-441a-8aef-ef8588e5fa7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:28.848950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3255c162-dacd-441a-8aef-ef8588e5fa7c admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.849236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3255c162-dacd-441a-8aef-ef8588e5fa7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.849511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3255c162-dacd-441a-8aef-ef8588e5fa7c admin admin] PATH_INFO: `/v3/projects/57894aeded144218a018791ead7f8623/users/125f81b1784642c2975be291ddd2996c/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.864336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3255c162-dacd-441a-8aef-ef8588e5fa7c admin admin] RBAC: Authorizing `identity:create_grant(project_id=57894aeded144218a018791ead7f8623, user_id=125f81b1784642c2975be291ddd2996c, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:28.867876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3255c162-dacd-441a-8aef-ef8588e5fa7c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:28.883531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1025/2057] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:28 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:28.896440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3520788-ab83-46f7-abfb-3696e2fff506 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:28.897904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a3520788-ab83-46f7-abfb-3696e2fff506 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:28.957969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1033/2058] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:28 2022] PUT /identity/v3/projects/57894aeded144218a018791ead7f8623/users/125f81b1784642c2975be291ddd2996c/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:28.967751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b393ee-9471-4881-8e86-fbff82ff9332 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:28.968050 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b393ee-9471-4881-8e86-fbff82ff9332 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:28.968336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b393ee-9471-4881-8e86-fbff82ff9332 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:28.986879 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3520788-ab83-46f7-abfb-3696e2fff506 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:29.100447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-57b393ee-9471-4881-8e86-fbff82ff9332 None None] MFA Rules not processed for user `125f81b1784642c2975be291ddd2996c`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:29.101840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-57b393ee-9471-4881-8e86-fbff82ff9332 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:29.109148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15385d92-6e9d-45d2-b795-fff88b1c2264 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a83b4eb626343f3aa5b1c9f3fc828fd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:29.109804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15385d92-6e9d-45d2-b795-fff88b1c2264 service glance] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:29.110085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15385d92-6e9d-45d2-b795-fff88b1c2264 service glance] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:29.110362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15385d92-6e9d-45d2-b795-fff88b1c2264 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:29.165264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1034/2059] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:28 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:29.169194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15385d92-6e9d-45d2-b795-fff88b1c2264 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:29.170215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b393ee-9471-4881-8e86-fbff82ff9332 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:29.170711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15385d92-6e9d-45d2-b795-fff88b1c2264 service glance] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:29.180721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1026/2060] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:28 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:06:29.236341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b393ee-9471-4881-8e86-fbff82ff9332 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:29.241936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c687dfdb-7503-4daf-9307-d1f622d1b692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:29.242680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c687dfdb-7503-4daf-9307-d1f622d1b692 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:29.243037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c687dfdb-7503-4daf-9307-d1f622d1b692 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:29.243383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c687dfdb-7503-4daf-9307-d1f622d1b692 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:29.244167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c687dfdb-7503-4daf-9307-d1f622d1b692 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:29.246848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c687dfdb-7503-4daf-9307-d1f622d1b692 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:29.357904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1035/2061] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:29 2022] POST /identity/v3/projects => generated 367 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:29.361930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15385d92-6e9d-45d2-b795-fff88b1c2264 service glance] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:29.373374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15385d92-6e9d-45d2-b795-fff88b1c2264 service glance] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:29.434205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7095ac-2e12-44cd-a905-958d8422a927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:29.434878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7095ac-2e12-44cd-a905-958d8422a927 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:29.435179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7095ac-2e12-44cd-a905-958d8422a927 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:29.435450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7095ac-2e12-44cd-a905-958d8422a927 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:29.436147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7095ac-2e12-44cd-a905-958d8422a927 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:29.443306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7095ac-2e12-44cd-a905-958d8422a927 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:29.514893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1027/2062] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:29 2022] POST /identity/v3/users => generated 290 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:29.519156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c687dfdb-7503-4daf-9307-d1f622d1b692 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:29.530355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c687dfdb-7503-4daf-9307-d1f622d1b692 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:29.535220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d23d680-4edd-4aa3-93ca-f06c031c2404 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:29.536119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d23d680-4edd-4aa3-93ca-f06c031c2404 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:29.536407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d23d680-4edd-4aa3-93ca-f06c031c2404 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:29.536688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d23d680-4edd-4aa3-93ca-f06c031c2404 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:29.537483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d23d680-4edd-4aa3-93ca-f06c031c2404 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:29.539121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d23d680-4edd-4aa3-93ca-f06c031c2404 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:29.550299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1036/2063] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:29.554491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7095ac-2e12-44cd-a905-958d8422a927 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:29.565459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7095ac-2e12-44cd-a905-958d8422a927 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:29.570534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410cab8c-21b1-4849-8ecf-076dac32c699 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:29.571429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410cab8c-21b1-4849-8ecf-076dac32c699 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:29.571714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410cab8c-21b1-4849-8ecf-076dac32c699 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:29.574096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410cab8c-21b1-4849-8ecf-076dac32c699 admin admin] PATH_INFO: `/v3/projects/a9e11e35ec834eeaa351b48acb4e3266/users/d58b2fed8d544fddb4815881f237b803/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:29.587512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410cab8c-21b1-4849-8ecf-076dac32c699 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9e11e35ec834eeaa351b48acb4e3266, user_id=d58b2fed8d544fddb4815881f237b803, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:29.590488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410cab8c-21b1-4849-8ecf-076dac32c699 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:29.727312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1028/2064] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:29 2022] PUT /identity/v3/projects/a9e11e35ec834eeaa351b48acb4e3266/users/d58b2fed8d544fddb4815881f237b803/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:29.728587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d23d680-4edd-4aa3-93ca-f06c031c2404 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:29.738781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d23d680-4edd-4aa3-93ca-f06c031c2404 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:29.774077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:29.797234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e7a07e-6479-4075-a58c-cb37f3f68235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:29.797918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7a07e-6479-4075-a58c-cb37f3f68235 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:29.798208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7a07e-6479-4075-a58c-cb37f3f68235 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:29.798490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7a07e-6479-4075-a58c-cb37f3f68235 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:29.799382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e7a07e-6479-4075-a58c-cb37f3f68235 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:29.801060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e7a07e-6479-4075-a58c-cb37f3f68235 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:29.817194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1037/2065] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:29 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:29.818858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410cab8c-21b1-4849-8ecf-076dac32c699 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:29.831284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410cab8c-21b1-4849-8ecf-076dac32c699 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:29.836231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e069e4e8-5c27-4639-b16d-28da8f837b9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:29.837210 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e069e4e8-5c27-4639-b16d-28da8f837b9e admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:29.837495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e069e4e8-5c27-4639-b16d-28da8f837b9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:29.837770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e069e4e8-5c27-4639-b16d-28da8f837b9e admin admin] PATH_INFO: `/v3/projects/a9e11e35ec834eeaa351b48acb4e3266/users/d58b2fed8d544fddb4815881f237b803/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:29.839791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e069e4e8-5c27-4639-b16d-28da8f837b9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9e11e35ec834eeaa351b48acb4e3266, user_id=d58b2fed8d544fddb4815881f237b803, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:29.842530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e069e4e8-5c27-4639-b16d-28da8f837b9e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:29.868500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1029/2066] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:29 2022] PUT /identity/v3/projects/a9e11e35ec834eeaa351b48acb4e3266/users/d58b2fed8d544fddb4815881f237b803/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:29.875215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3b4475-16ea-4435-b869-d91ef570477e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:29.875526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3b4475-16ea-4435-b869-d91ef570477e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:29.875778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3b4475-16ea-4435-b869-d91ef570477e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:29.930461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:29.934378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4e3b4475-16ea-4435-b869-d91ef570477e None None] MFA Rules not processed for user `d58b2fed8d544fddb4815881f237b803`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:29.935617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4e3b4475-16ea-4435-b869-d91ef570477e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:30.042759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1038/2067] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:29 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:30.054192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e069e4e8-5c27-4639-b16d-28da8f837b9e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:30.055998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e069e4e8-5c27-4639-b16d-28da8f837b9e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:30.119662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e069e4e8-5c27-4639-b16d-28da8f837b9e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:30.125350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f315133e-b729-4659-8874-4f047d837694 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:30.126019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f315133e-b729-4659-8874-4f047d837694 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:30.126303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f315133e-b729-4659-8874-4f047d837694 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:30.126583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f315133e-b729-4659-8874-4f047d837694 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:30.138580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f315133e-b729-4659-8874-4f047d837694 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:30.140130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f315133e-b729-4659-8874-4f047d837694 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:30.151233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1030/2068] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:30 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:31.113847 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:31.810858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:34.363562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3b4475-16ea-4435-b869-d91ef570477e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:34.501600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3b4475-16ea-4435-b869-d91ef570477e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:34.520444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:34.521425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:34.525456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:34.525456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] PATH_INFO: `/v3/users/125f81b1784642c2975be291ddd2996c` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:34.532508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] RBAC: Authorizing `identity:delete_user(user_id=125f81b1784642c2975be291ddd2996c)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:34.537241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:34.651898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:34.805181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '125f81b1784642c2975be291ddd2996c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:34.807672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '125f81b1784642c2975be291ddd2996c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:34.845119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '125f81b1784642c2975be291ddd2996c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:34.965454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '125f81b1784642c2975be291ddd2996c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:34.966430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '125f81b1784642c2975be291ddd2996c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:34.999788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1039/2069] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:34 2022] DELETE /identity/v3/users/125f81b1784642c2975be291ddd2996c => generated 0 bytes in 636 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:35.004005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b35b0d-8e09-4478-9b0f-bc1fc103e362 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:35.005058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b35b0d-8e09-4478-9b0f-bc1fc103e362 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:35.005509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75b35b0d-8e09-4478-9b0f-bc1fc103e362 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:35.034159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-75b35b0d-8e09-4478-9b0f-bc1fc103e362 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:35.040492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-75b35b0d-8e09-4478-9b0f-bc1fc103e362 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:35.148260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1031/2070] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:35.178979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:35.180787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:35.204653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dcf6ca-f84a-4d78-b318-7d7e9f4dbc6e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:35.205398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dcf6ca-f84a-4d78-b318-7d7e9f4dbc6e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:35.205685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dcf6ca-f84a-4d78-b318-7d7e9f4dbc6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:35.258770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e9dcf6ca-f84a-4d78-b318-7d7e9f4dbc6e None None] MFA Rules not processed for user `25f73d0bd4e64057a922d11be46eb04c`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:35.260030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e9dcf6ca-f84a-4d78-b318-7d7e9f4dbc6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:35.301479 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c14c4f-7f6b-4ff7-a99b-a6a91491a830 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:35.306173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5743b0a1-dd34-4a3f-9e33-1b2bc5c93a82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:35.306942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743b0a1-dd34-4a3f-9e33-1b2bc5c93a82 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:35.307303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743b0a1-dd34-4a3f-9e33-1b2bc5c93a82 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:35.307717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5743b0a1-dd34-4a3f-9e33-1b2bc5c93a82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:35.322601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5743b0a1-dd34-4a3f-9e33-1b2bc5c93a82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:35.324509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5743b0a1-dd34-4a3f-9e33-1b2bc5c93a82 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:35.342771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1040/2071] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:35.372627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1032/2072] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:35 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:35.383855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c439f4a-795e-4445-8e15-f1dae9402f61 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:35.384388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c439f4a-795e-4445-8e15-f1dae9402f61 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:35.384388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c439f4a-795e-4445-8e15-f1dae9402f61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:35.412663 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6c439f4a-795e-4445-8e15-f1dae9402f61 None None] MFA Rules not processed for user `25f73d0bd4e64057a922d11be46eb04c`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:35.414395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6c439f4a-795e-4445-8e15-f1dae9402f61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:35.431576 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1041/2073] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:35 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:35.445684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dcf6ca-f84a-4d78-b318-7d7e9f4dbc6e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:35.447069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e9dcf6ca-f84a-4d78-b318-7d7e9f4dbc6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:35.578977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dcf6ca-f84a-4d78-b318-7d7e9f4dbc6e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:35.588805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60df23f1-9d5a-406e-9cdb-bebfb499ea42 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:35.589308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df23f1-9d5a-406e-9cdb-bebfb499ea42 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:35.590309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df23f1-9d5a-406e-9cdb-bebfb499ea42 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:35.591262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60df23f1-9d5a-406e-9cdb-bebfb499ea42 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:35.615092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c439f4a-795e-4445-8e15-f1dae9402f61 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:35.617067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6c439f4a-795e-4445-8e15-f1dae9402f61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:35.624802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60df23f1-9d5a-406e-9cdb-bebfb499ea42 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:35.627862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60df23f1-9d5a-406e-9cdb-bebfb499ea42 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:35.633502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c439f4a-795e-4445-8e15-f1dae9402f61 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:35.637891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:35.639239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:35.640237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:35.640525 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:35.640813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] PATH_INFO: `/v3/users/9b0e50775367405c8d997036dd33aac7` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:35.642380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b0e50775367405c8d997036dd33aac7)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:35.644467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:35.646040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1033/2074] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:35 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:35.953965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60df23f1-9d5a-406e-9cdb-bebfb499ea42 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:35.955755 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-60df23f1-9d5a-406e-9cdb-bebfb499ea42 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:35.984537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b0e50775367405c8d997036dd33aac7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:35.984693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60df23f1-9d5a-406e-9cdb-bebfb499ea42 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:35.985295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b0e50775367405c8d997036dd33aac7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:35.997558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:35.998501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:35.998823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:35.999067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] PATH_INFO: `/v3/projects/57894aeded144218a018791ead7f8623` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:36.000476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=57894aeded144218a018791ead7f8623)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:36.001580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b0e50775367405c8d997036dd33aac7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.007459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:36.130347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b0e50775367405c8d997036dd33aac7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.131965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6dfa664-3aa0-4ff6-a088-88d33de956be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b0e50775367405c8d997036dd33aac7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.156393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1042/2075] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:35 2022] DELETE /identity/v3/users/9b0e50775367405c8d997036dd33aac7 => generated 0 bytes in 541 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:36.161971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce315178-9f4f-4c6c-b9c2-229764fbb3ab None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:36.162570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce315178-9f4f-4c6c-b9c2-229764fbb3ab None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:36.163163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce315178-9f4f-4c6c-b9c2-229764fbb3ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:36.187613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ce315178-9f4f-4c6c-b9c2-229764fbb3ab None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:36.189128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ce315178-9f4f-4c6c-b9c2-229764fbb3ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:36.212007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57894aeded144218a018791ead7f8623', 'request_id': 'req-472546d7-2dc6-47a4-a58e-28598ca6d5bb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.278543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1043/2076] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:36.291497 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce315178-9f4f-4c6c-b9c2-229764fbb3ab None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:36.293328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ce315178-9f4f-4c6c-b9c2-229764fbb3ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:36.343239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57894aeded144218a018791ead7f8623', 'request_id': 'req-472546d7-2dc6-47a4-a58e-28598ca6d5bb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.354279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57894aeded144218a018791ead7f8623', 'request_id': 'req-472546d7-2dc6-47a4-a58e-28598ca6d5bb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.367952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] The token cache is being invalidate because project 57894aeded144218a018791ead7f8623 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:36.368650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.370264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.373790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1034/2077] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:35 2022] DELETE /identity/v3/projects/57894aeded144218a018791ead7f8623 => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:36.377889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:36.379977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:36.416029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce315178-9f4f-4c6c-b9c2-229764fbb3ab None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:36.499494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472546d7-2dc6-47a4-a58e-28598ca6d5bb admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:36.506925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:36.508095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:36.508750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:36.509528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] PATH_INFO: `/v3/users/d58b2fed8d544fddb4815881f237b803` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:36.511365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d58b2fed8d544fddb4815881f237b803)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:36.514862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:36.538708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:36.540134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:36.540426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:36.540870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:36.543554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:36.549690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:36.585192 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:36.590212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:36.606027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1044/2078] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:36.610453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:36.612328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:36.626411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d85dc97-4295-47b7-ae4f-5b179c93f8d6 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:36.631641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:36.632573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:36.632878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:36.633324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] PATH_INFO: `/v3/users/dd015e885aec411cb4eef64f5934d68d` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:36.635258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd015e885aec411cb4eef64f5934d68d)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:36.638752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:36.725999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd58b2fed8d544fddb4815881f237b803'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.726747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd58b2fed8d544fddb4815881f237b803'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.727469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd58b2fed8d544fddb4815881f237b803'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.736557 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd58b2fed8d544fddb4815881f237b803'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.873473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f1768f4b-e469-4467-89f0-1605328ac950 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd58b2fed8d544fddb4815881f237b803'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.895078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1035/2079] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:36 2022] DELETE /identity/v3/users/d58b2fed8d544fddb4815881f237b803 => generated 0 bytes in 517 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:36.896527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a6de79-5857-4428-99b7-88cc8a9ea610 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:36.896955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a6de79-5857-4428-99b7-88cc8a9ea610 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:36.897144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a6de79-5857-4428-99b7-88cc8a9ea610 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:36.908080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd015e885aec411cb4eef64f5934d68d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.909222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd015e885aec411cb4eef64f5934d68d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.919339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd015e885aec411cb4eef64f5934d68d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:36.939326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:36.954467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-17a6de79-5857-4428-99b7-88cc8a9ea610 None None] MFA Rules not processed for user `25f73d0bd4e64057a922d11be46eb04c`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:36.954467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-17a6de79-5857-4428-99b7-88cc8a9ea610 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:37.036980 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1036/2080] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:36 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:37.045278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd015e885aec411cb4eef64f5934d68d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:37.046032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd015e885aec411cb4eef64f5934d68d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:37.076881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1045/2081] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:36 2022] DELETE /identity/v3/users/dd015e885aec411cb4eef64f5934d68d => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:37.081214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970b60fd-d839-4574-b40a-6a0e41cd3140 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:37.081536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970b60fd-d839-4574-b40a-6a0e41cd3140 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:37.085510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970b60fd-d839-4574-b40a-6a0e41cd3140 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:37.103917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-970b60fd-d839-4574-b40a-6a0e41cd3140 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:37.106450 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-970b60fd-d839-4574-b40a-6a0e41cd3140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:37.204676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1037/2082] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:37.218228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:37.220930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:37.320044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82bd9e2-2661-4125-9915-46cbc6683659 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:37.324443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-275700e1-7a81-4c32-b502-3d9eee811955 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:37.325287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275700e1-7a81-4c32-b502-3d9eee811955 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:37.325679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275700e1-7a81-4c32-b502-3d9eee811955 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:37.326062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-275700e1-7a81-4c32-b502-3d9eee811955 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:37.337291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275700e1-7a81-4c32-b502-3d9eee811955 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:37.338718 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-275700e1-7a81-4c32-b502-3d9eee811955 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:37.349337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1046/2083] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:37.424277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970b60fd-d839-4574-b40a-6a0e41cd3140 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:37.425798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-970b60fd-d839-4574-b40a-6a0e41cd3140 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:37.438910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970b60fd-d839-4574-b40a-6a0e41cd3140 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:37.452002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:37.453353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:37.454375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:37.454808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] PATH_INFO: `/v3/projects/a40832ec7fa34f71b4657fc372672f41` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:37.456581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a40832ec7fa34f71b4657fc372672f41)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:37.458965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:37.628641 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a40832ec7fa34f71b4657fc372672f41', 'request_id': 'req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:37.690515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a40832ec7fa34f71b4657fc372672f41', 'request_id': 'req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:37.716875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a40832ec7fa34f71b4657fc372672f41', 'request_id': 'req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:37.746665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] The token cache is being invalidate because project a40832ec7fa34f71b4657fc372672f41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:37.747013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:37.749491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ef02e51d-2a54-4ab4-9d44-c79cfbf86615 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:37.750203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1038/2084] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:37 2022] DELETE /identity/v3/projects/a40832ec7fa34f71b4657fc372672f41 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:37.756189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9b71a3-8b14-4b69-8e91-f99b8282d728 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:37.758695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9b71a3-8b14-4b69-8e91-f99b8282d728 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:37.759277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab9b71a3-8b14-4b69-8e91-f99b8282d728 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:37.811040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ab9b71a3-8b14-4b69-8e91-f99b8282d728 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:37.814619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ab9b71a3-8b14-4b69-8e91-f99b8282d728 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:37.943839 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1047/2085] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:37.945473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f53ada-4e7b-46cd-9d77-77ec31901cf9 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:37.945721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f53ada-4e7b-46cd-9d77-77ec31901cf9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:37.946009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f53ada-4e7b-46cd-9d77-77ec31901cf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:38.001483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-24f53ada-4e7b-46cd-9d77-77ec31901cf9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:38.002580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-24f53ada-4e7b-46cd-9d77-77ec31901cf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:38.025527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1039/2086] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:38.030409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9b71a3-8b14-4b69-8e91-f99b8282d728 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:38.041009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab9b71a3-8b14-4b69-8e91-f99b8282d728 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:38.047466 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b9f337-53a7-409c-9611-72bd4f489a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:38.047895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b9f337-53a7-409c-9611-72bd4f489a59 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:38.048181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b9f337-53a7-409c-9611-72bd4f489a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:38.048459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81b9f337-53a7-409c-9611-72bd4f489a59 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:38.049248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b9f337-53a7-409c-9611-72bd4f489a59 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:38.049556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b9f337-53a7-409c-9611-72bd4f489a59 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:38.051345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81b9f337-53a7-409c-9611-72bd4f489a59 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:38.079476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1048/2087] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:06:38.090913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f53ada-4e7b-46cd-9d77-77ec31901cf9 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:38.110292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f53ada-4e7b-46cd-9d77-77ec31901cf9 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:38.117346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19953ff-3004-4ffc-a24e-8cc2856e3771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:38.118251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19953ff-3004-4ffc-a24e-8cc2856e3771 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:38.118809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19953ff-3004-4ffc-a24e-8cc2856e3771 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:38.119299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19953ff-3004-4ffc-a24e-8cc2856e3771 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:38.122040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19953ff-3004-4ffc-a24e-8cc2856e3771 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:38.124454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19953ff-3004-4ffc-a24e-8cc2856e3771 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:38.303431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1040/2088] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:38 2022] POST /identity/v3/projects => generated 373 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:38.307601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b9f337-53a7-409c-9611-72bd4f489a59 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:38.333588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19953ff-3004-4ffc-a24e-8cc2856e3771 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:38.341995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81b9f337-53a7-409c-9611-72bd4f489a59 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:38.347906 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c19953ff-3004-4ffc-a24e-8cc2856e3771 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:38.467617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19953ff-3004-4ffc-a24e-8cc2856e3771 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:38.473462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607ecdc-8e3a-4572-98c3-59dfcb268d63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:38.474781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:38.475107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607ecdc-8e3a-4572-98c3-59dfcb268d63 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:38.475541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607ecdc-8e3a-4572-98c3-59dfcb268d63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:38.475667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:38.477078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:38.477680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] PATH_INFO: `/v3/projects/a9e11e35ec834eeaa351b48acb4e3266` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:38.482486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9e11e35ec834eeaa351b48acb4e3266)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:38.484892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:38.486151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e607ecdc-8e3a-4572-98c3-59dfcb268d63 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:38.486999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607ecdc-8e3a-4572-98c3-59dfcb268d63 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:38.489714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e607ecdc-8e3a-4572-98c3-59dfcb268d63 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:38.576298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1049/2089] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:38 2022] POST /identity/v3/users => generated 287 bytes in 269 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:38.577621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607ecdc-8e3a-4572-98c3-59dfcb268d63 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:38.579454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e607ecdc-8e3a-4572-98c3-59dfcb268d63 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:38.591896 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e607ecdc-8e3a-4572-98c3-59dfcb268d63 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:38.596545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:38.597463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:38.597766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:38.598072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] PATH_INFO: `/v3/projects/7664faa3fbfe40deaf65d87e26cccab6` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:38.599315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7664faa3fbfe40deaf65d87e26cccab6)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:38.601298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:38.707491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9e11e35ec834eeaa351b48acb4e3266', 'request_id': 'req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.790562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7664faa3fbfe40deaf65d87e26cccab6', 'request_id': 'req-8a11364d-159e-47b9-a7ec-f5f56659c2d2'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.798066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7664faa3fbfe40deaf65d87e26cccab6', 'request_id': 'req-8a11364d-159e-47b9-a7ec-f5f56659c2d2'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.804382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7664faa3fbfe40deaf65d87e26cccab6', 'request_id': 'req-8a11364d-159e-47b9-a7ec-f5f56659c2d2'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.844000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9e11e35ec834eeaa351b48acb4e3266', 'request_id': 'req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.850872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9e11e35ec834eeaa351b48acb4e3266', 'request_id': 'req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.882742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] The token cache is being invalidate because project a9e11e35ec834eeaa351b48acb4e3266 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:38.883133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.883984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.885755 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1041/2090] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:38 2022] DELETE /identity/v3/projects/a9e11e35ec834eeaa351b48acb4e3266 => generated 0 bytes in 553 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:38.890596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:38.891993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:38.972252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] The token cache is being invalidate because project 7664faa3fbfe40deaf65d87e26cccab6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:38.981473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.982341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a11364d-159e-47b9-a7ec-f5f56659c2d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:38.984703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1050/2091] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:38 2022] DELETE /identity/v3/projects/7664faa3fbfe40deaf65d87e26cccab6 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:38.986645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b400223-7145-44ad-9a14-cc906fbf3624 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:38.987128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b400223-7145-44ad-9a14-cc906fbf3624 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:38.988175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b400223-7145-44ad-9a14-cc906fbf3624 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.031234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0d9aaba-b975-4ebd-8ac9-3af835b4959c admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:39.060691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3b400223-7145-44ad-9a14-cc906fbf3624 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:39.061961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3b400223-7145-44ad-9a14-cc906fbf3624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:39.107164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e0361d-d874-46e4-920a-f1c127623a78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:39.107164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e0361d-d874-46e4-920a-f1c127623a78 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.107164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e0361d-d874-46e4-920a-f1c127623a78 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.107164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73e0361d-d874-46e4-920a-f1c127623a78 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.107864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e0361d-d874-46e4-920a-f1c127623a78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:39.108412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73e0361d-d874-46e4-920a-f1c127623a78 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:39.119032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1042/2092] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:38 2022] GET /identity/v3/roles => generated 1420 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:39.120130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e0361d-d874-46e4-920a-f1c127623a78 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:39.122921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-73e0361d-d874-46e4-920a-f1c127623a78 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:39.136963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73e0361d-d874-46e4-920a-f1c127623a78 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:39.148049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1051/2093] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:39.148885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:39.149484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b400223-7145-44ad-9a14-cc906fbf3624 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:39.153069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.153069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.153069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] PATH_INFO: `/v3/users/d9a69c2359bc449f8ebca59bc3be4266` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.153324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3b400223-7145-44ad-9a14-cc906fbf3624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:39.153399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9a69c2359bc449f8ebca59bc3be4266)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:39.155294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:39.169649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b400223-7145-44ad-9a14-cc906fbf3624 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:39.174760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-684099f5-ad5c-45d1-9a55-ba372c607ef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:39.175438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684099f5-ad5c-45d1-9a55-ba372c607ef3 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.175723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684099f5-ad5c-45d1-9a55-ba372c607ef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.176001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-684099f5-ad5c-45d1-9a55-ba372c607ef3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.176734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684099f5-ad5c-45d1-9a55-ba372c607ef3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:39.179227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-684099f5-ad5c-45d1-9a55-ba372c607ef3 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:39.191604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1052/2094] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:39 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:39.192176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d0129a-7153-4576-8ea4-cf28aa8f84e5 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.192604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d0129a-7153-4576-8ea4-cf28aa8f84e5 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.192918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d0129a-7153-4576-8ea4-cf28aa8f84e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.213683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-32d0129a-7153-4576-8ea4-cf28aa8f84e5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:39.217023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-32d0129a-7153-4576-8ea4-cf28aa8f84e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:39.227992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1053/2095] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:39.229437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d0129a-7153-4576-8ea4-cf28aa8f84e5 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:39.243643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d0129a-7153-4576-8ea4-cf28aa8f84e5 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:39.250820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87366cd3-88d2-49ec-a960-573208d0eded admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:39.252359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87366cd3-88d2-49ec-a960-573208d0eded admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.252640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87366cd3-88d2-49ec-a960-573208d0eded admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.252952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87366cd3-88d2-49ec-a960-573208d0eded admin admin] PATH_INFO: `/v3/projects/8ec25481febb4747bcb07175347ed0bb/users/8e92014bd6c446f1bcbf31f475ca7092/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.291156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87366cd3-88d2-49ec-a960-573208d0eded admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ec25481febb4747bcb07175347ed0bb, user_id=8e92014bd6c446f1bcbf31f475ca7092, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:39.294236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87366cd3-88d2-49ec-a960-573208d0eded admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:39.430608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1054/2096] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:39 2022] PUT /identity/v3/projects/8ec25481febb4747bcb07175347ed0bb/users/8e92014bd6c446f1bcbf31f475ca7092/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:39.431739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87366cd3-88d2-49ec-a960-573208d0eded admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:39.434102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9a69c2359bc449f8ebca59bc3be4266'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:39.435647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9a69c2359bc449f8ebca59bc3be4266'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:39.436383 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9a69c2359bc449f8ebca59bc3be4266'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:39.451970 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9a69c2359bc449f8ebca59bc3be4266'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:39.531591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87366cd3-88d2-49ec-a960-573208d0eded admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:39.538697 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19319eae-e2ae-4d36-8aa2-99a7540acdab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:39.539625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19319eae-e2ae-4d36-8aa2-99a7540acdab admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.539980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19319eae-e2ae-4d36-8aa2-99a7540acdab admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.545671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19319eae-e2ae-4d36-8aa2-99a7540acdab admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.545671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19319eae-e2ae-4d36-8aa2-99a7540acdab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:39.545671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19319eae-e2ae-4d36-8aa2-99a7540acdab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:39.545671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19319eae-e2ae-4d36-8aa2-99a7540acdab admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:39.562996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1055/2097] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:06:39.564131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9a69c2359bc449f8ebca59bc3be4266'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:39.564278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2609e7-543e-49cb-9526-7895f557055a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.564484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2609e7-543e-49cb-9526-7895f557055a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.564795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f2609e7-543e-49cb-9526-7895f557055a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.586954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1043/2098] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:39 2022] DELETE /identity/v3/users/d9a69c2359bc449f8ebca59bc3be4266 => generated 0 bytes in 467 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:39.588049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:39.592994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6f2609e7-543e-49cb-9526-7895f557055a None None] MFA Rules not processed for user `8e92014bd6c446f1bcbf31f475ca7092`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:39.594216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6f2609e7-543e-49cb-9526-7895f557055a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:39.598477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:39.716648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1056/2099] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:39 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:39.726731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b8b6d42-8bec-4472-9b59-b919c295c43a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:39.731858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2609e7-543e-49cb-9526-7895f557055a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:39.738511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6f2609e7-543e-49cb-9526-7895f557055a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:39.741987 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27fa017-9b4c-45e5-b289-a648251de1ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:39.743005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27fa017-9b4c-45e5-b289-a648251de1ea admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.744506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27fa017-9b4c-45e5-b289-a648251de1ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.744926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27fa017-9b4c-45e5-b289-a648251de1ea admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.746480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27fa017-9b4c-45e5-b289-a648251de1ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:39.749263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27fa017-9b4c-45e5-b289-a648251de1ea admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:39.807145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f2609e7-543e-49cb-9526-7895f557055a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:39.812592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062c3c47-a879-4fc2-bb86-65f0827c4ce0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:39.812592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062c3c47-a879-4fc2-bb86-65f0827c4ce0 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.812592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062c3c47-a879-4fc2-bb86-65f0827c4ce0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.812592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062c3c47-a879-4fc2-bb86-65f0827c4ce0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:39.825444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062c3c47-a879-4fc2-bb86-65f0827c4ce0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:39.826987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062c3c47-a879-4fc2-bb86-65f0827c4ce0 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:39.838855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1057/2100] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:39 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:39.872437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062c3c47-a879-4fc2-bb86-65f0827c4ce0 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:39.874890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-062c3c47-a879-4fc2-bb86-65f0827c4ce0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:39.876007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1044/2101] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:39 2022] POST /identity/v3/projects => generated 377 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:39.881284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27fa017-9b4c-45e5-b289-a648251de1ea admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:39.893936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062c3c47-a879-4fc2-bb86-65f0827c4ce0 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:39.997145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:39.998179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:39.998541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:39.998880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] PATH_INFO: `/v3/users/8e92014bd6c446f1bcbf31f475ca7092` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.000656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e92014bd6c446f1bcbf31f475ca7092)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:40.002797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:40.048715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27fa017-9b4c-45e5-b289-a648251de1ea admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:40.056041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10c2546-852f-416c-a832-302c2114ed75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:40.057594 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10c2546-852f-416c-a832-302c2114ed75 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:40.058577 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10c2546-852f-416c-a832-302c2114ed75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:40.059069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10c2546-852f-416c-a832-302c2114ed75 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.060314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10c2546-852f-416c-a832-302c2114ed75 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:40.063779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10c2546-852f-416c-a832-302c2114ed75 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:40.399277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1045/2102] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:39 2022] POST /identity/v3/users => generated 289 bytes in 520 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:40.405819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10c2546-852f-416c-a832-302c2114ed75 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:40.417921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10c2546-852f-416c-a832-302c2114ed75 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:40.420928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e92014bd6c446f1bcbf31f475ca7092'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.421921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e92014bd6c446f1bcbf31f475ca7092'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.423666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3650dd4-0c04-40b8-973e-d4f925e8b7db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:40.424927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3650dd4-0c04-40b8-973e-d4f925e8b7db admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:40.425292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3650dd4-0c04-40b8-973e-d4f925e8b7db admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:40.425639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3650dd4-0c04-40b8-973e-d4f925e8b7db admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.426793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3650dd4-0c04-40b8-973e-d4f925e8b7db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:40.428162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3650dd4-0c04-40b8-973e-d4f925e8b7db admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:40.452007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e92014bd6c446f1bcbf31f475ca7092'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.475566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1046/2103] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:40 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:40.476451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3650dd4-0c04-40b8-973e-d4f925e8b7db admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:40.484487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e3650dd4-0c04-40b8-973e-d4f925e8b7db admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:40.515770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3650dd4-0c04-40b8-973e-d4f925e8b7db admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:40.521077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:40.522378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:40.522829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:40.523374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] PATH_INFO: `/v3/projects/38c6e46ec17c46b4ae05a4edc5099e8e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.524696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] RBAC: Authorizing `identity:delete_project(project_id=38c6e46ec17c46b4ae05a4edc5099e8e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:40.526848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:40.582490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e92014bd6c446f1bcbf31f475ca7092'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.583174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e92014bd6c446f1bcbf31f475ca7092'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.620418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1058/2104] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:39 2022] DELETE /identity/v3/users/8e92014bd6c446f1bcbf31f475ca7092 => generated 0 bytes in 745 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:40.620418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:40.626064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:40.695689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767c310c-829a-4366-a30a-18de71c8307d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:40.700395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601c8f95-7984-4ab7-b4e1-088a99d83caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:40.701096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601c8f95-7984-4ab7-b4e1-088a99d83caa admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:40.701387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601c8f95-7984-4ab7-b4e1-088a99d83caa admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:40.701667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601c8f95-7984-4ab7-b4e1-088a99d83caa admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.702394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601c8f95-7984-4ab7-b4e1-088a99d83caa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:40.704016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-601c8f95-7984-4ab7-b4e1-088a99d83caa admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:40.712106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38c6e46ec17c46b4ae05a4edc5099e8e', 'request_id': 'req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.714239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1059/2105] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:40 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:40.714239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34257cc1-214a-4409-8a79-d6d7d777d6d4 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:40.714239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34257cc1-214a-4409-8a79-d6d7d777d6d4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:40.714239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34257cc1-214a-4409-8a79-d6d7d777d6d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.733016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-34257cc1-214a-4409-8a79-d6d7d777d6d4 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:40.734222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-34257cc1-214a-4409-8a79-d6d7d777d6d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:40.794244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1060/2106] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:40.795098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34257cc1-214a-4409-8a79-d6d7d777d6d4 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:40.810199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34257cc1-214a-4409-8a79-d6d7d777d6d4 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:40.814856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccf42e15-10b1-44a5-a377-534908f78812 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:40.815745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf42e15-10b1-44a5-a377-534908f78812 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:40.816030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf42e15-10b1-44a5-a377-534908f78812 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:40.816325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf42e15-10b1-44a5-a377-534908f78812 admin admin] PATH_INFO: `/v3/projects/efa7b57104734551afbc94c410a21dad/users/dccc9b250645451c94b15aae40beff0f/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.832507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38c6e46ec17c46b4ae05a4edc5099e8e', 'request_id': 'req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.835688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf42e15-10b1-44a5-a377-534908f78812 admin admin] RBAC: Authorizing `identity:create_grant(project_id=efa7b57104734551afbc94c410a21dad, user_id=dccc9b250645451c94b15aae40beff0f, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:40.838449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccf42e15-10b1-44a5-a377-534908f78812 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:40.842035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38c6e46ec17c46b4ae05a4edc5099e8e', 'request_id': 'req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.854330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] The token cache is being invalidate because project 38c6e46ec17c46b4ae05a4edc5099e8e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:40.854760 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.856058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:40.857952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1047/2107] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:40 2022] DELETE /identity/v3/projects/38c6e46ec17c46b4ae05a4edc5099e8e => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:40.858737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:40.860149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:40.933276 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6b2d54-c9e4-4e19-8c11-45e714c2923c admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:40.937048 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82bb4725-cc51-468d-a4f5-61732a741191 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:40.937700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bb4725-cc51-468d-a4f5-61732a741191 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:40.937985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bb4725-cc51-468d-a4f5-61732a741191 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:40.938259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82bb4725-cc51-468d-a4f5-61732a741191 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.940633 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-82bb4725-cc51-468d-a4f5-61732a741191 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:40.954008 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bb4725-cc51-468d-a4f5-61732a741191 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:40.955472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82bb4725-cc51-468d-a4f5-61732a741191 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:40.966926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1048/2108] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:40.968380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54118c8c-2385-44fe-99d7-0c119a9efb38 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:40.968672 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54118c8c-2385-44fe-99d7-0c119a9efb38 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:40.971915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54118c8c-2385-44fe-99d7-0c119a9efb38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.973372 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1061/2109] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:40 2022] PUT /identity/v3/projects/efa7b57104734551afbc94c410a21dad/users/dccc9b250645451c94b15aae40beff0f/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:40.984453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e144c70-d0e0-4e5e-bc44-4ed7d6cbb1f2 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:40.984806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e144c70-d0e0-4e5e-bc44-4ed7d6cbb1f2 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:40.985540 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e144c70-d0e0-4e5e-bc44-4ed7d6cbb1f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:40.994720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-54118c8c-2385-44fe-99d7-0c119a9efb38 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:40.996476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-54118c8c-2385-44fe-99d7-0c119a9efb38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:41.027413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-7e144c70-d0e0-4e5e-bc44-4ed7d6cbb1f2 None None] MFA Rules not processed for user `dccc9b250645451c94b15aae40beff0f`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:41.028759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7e144c70-d0e0-4e5e-bc44-4ed7d6cbb1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:41.074288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1049/2110] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:41.088752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c0ef66-11d1-4467-b1ec-4a8c9056ef2d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.089267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c0ef66-11d1-4467-b1ec-4a8c9056ef2d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.089689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c0ef66-11d1-4467-b1ec-4a8c9056ef2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.110659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a9c0ef66-11d1-4467-b1ec-4a8c9056ef2d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:41.112430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a9c0ef66-11d1-4467-b1ec-4a8c9056ef2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:41.122759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1050/2111] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:41.126820 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c0ef66-11d1-4467-b1ec-4a8c9056ef2d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.137222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c0ef66-11d1-4467-b1ec-4a8c9056ef2d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.142160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135118a6-5958-47b8-ac93-e556c5393137 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.142930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135118a6-5958-47b8-ac93-e556c5393137 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.144056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135118a6-5958-47b8-ac93-e556c5393137 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.144056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135118a6-5958-47b8-ac93-e556c5393137 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.144303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135118a6-5958-47b8-ac93-e556c5393137 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:41.144592 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135118a6-5958-47b8-ac93-e556c5393137 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.146250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135118a6-5958-47b8-ac93-e556c5393137 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.151826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1062/2112] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:40 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:41.156294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e144c70-d0e0-4e5e-bc44-4ed7d6cbb1f2 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.157751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7e144c70-d0e0-4e5e-bc44-4ed7d6cbb1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:41.160007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1051/2113] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:06:41.163955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135118a6-5958-47b8-ac93-e556c5393137 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.175658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135118a6-5958-47b8-ac93-e556c5393137 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.181340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4f8bf1-52da-4e75-ba52-d43d36de144d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.182796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4f8bf1-52da-4e75-ba52-d43d36de144d admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.183790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4f8bf1-52da-4e75-ba52-d43d36de144d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.184813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4f8bf1-52da-4e75-ba52-d43d36de144d admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.185254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4f8bf1-52da-4e75-ba52-d43d36de144d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.186246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e144c70-d0e0-4e5e-bc44-4ed7d6cbb1f2 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.188255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4f8bf1-52da-4e75-ba52-d43d36de144d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.194058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a94628-ea86-4ac2-82f9-7c03d90371d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.195181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a94628-ea86-4ac2-82f9-7c03d90371d9 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.195374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a94628-ea86-4ac2-82f9-7c03d90371d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.195730 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a94628-ea86-4ac2-82f9-7c03d90371d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.196824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a94628-ea86-4ac2-82f9-7c03d90371d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.198806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a94628-ea86-4ac2-82f9-7c03d90371d9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.226864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:41.283043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1052/2114] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:41 2022] POST /identity/v3/projects => generated 399 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:41.291183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4f8bf1-52da-4e75-ba52-d43d36de144d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.314991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1063/2115] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:41 2022] POST /identity/v3/projects => generated 377 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:41.337910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a94628-ea86-4ac2-82f9-7c03d90371d9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.398521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:41.442328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4f8bf1-52da-4e75-ba52-d43d36de144d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.479642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a94628-ea86-4ac2-82f9-7c03d90371d9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.487038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5ee5b2-e644-4087-b1f4-4592ce2b3edf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.487587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5ee5b2-e644-4087-b1f4-4592ce2b3edf admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.487962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5ee5b2-e644-4087-b1f4-4592ce2b3edf admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.488277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5ee5b2-e644-4087-b1f4-4592ce2b3edf admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.490547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5ee5b2-e644-4087-b1f4-4592ce2b3edf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.493321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5ee5b2-e644-4087-b1f4-4592ce2b3edf admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.561385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50875625-db44-42cb-844a-f4a7a54e76eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.562315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50875625-db44-42cb-844a-f4a7a54e76eb admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.565472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50875625-db44-42cb-844a-f4a7a54e76eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.565472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50875625-db44-42cb-844a-f4a7a54e76eb admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.565472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50875625-db44-42cb-844a-f4a7a54e76eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.566282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50875625-db44-42cb-844a-f4a7a54e76eb admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.652136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1064/2116] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:41 2022] POST /identity/v3/users => generated 289 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:41.652136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5ee5b2-e644-4087-b1f4-4592ce2b3edf admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.656243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1b5ee5b2-e644-4087-b1f4-4592ce2b3edf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:41.706079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1053/2117] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:41 2022] POST /identity/v3/users => generated 300 bytes in 417 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:41.713198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50875625-db44-42cb-844a-f4a7a54e76eb admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.724801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50875625-db44-42cb-844a-f4a7a54e76eb admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.732569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533ec16e-85d4-47f1-99c9-2dbe16c8e818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.733531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533ec16e-85d4-47f1-99c9-2dbe16c8e818 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.733926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533ec16e-85d4-47f1-99c9-2dbe16c8e818 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.734366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533ec16e-85d4-47f1-99c9-2dbe16c8e818 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.735593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533ec16e-85d4-47f1-99c9-2dbe16c8e818 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.737661 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533ec16e-85d4-47f1-99c9-2dbe16c8e818 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.742922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5ee5b2-e644-4087-b1f4-4592ce2b3edf admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.752159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1054/2118] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:41 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:41.753358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533ec16e-85d4-47f1-99c9-2dbe16c8e818 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.757632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.760566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.766395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.767409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] PATH_INFO: `/v3/projects/8ec25481febb4747bcb07175347ed0bb` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.767954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533ec16e-85d4-47f1-99c9-2dbe16c8e818 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.770754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ec25481febb4747bcb07175347ed0bb)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.775564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d8dcd2-24a8-426e-b74e-d90e12c7402c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.777153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d8dcd2-24a8-426e-b74e-d90e12c7402c admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.777585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d8dcd2-24a8-426e-b74e-d90e12c7402c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.778121 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d8dcd2-24a8-426e-b74e-d90e12c7402c admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.779167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d8dcd2-24a8-426e-b74e-d90e12c7402c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.781321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d8dcd2-24a8-426e-b74e-d90e12c7402c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.790146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.807214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1055/2119] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:41 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:41.807214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d8dcd2-24a8-426e-b74e-d90e12c7402c admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.814153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d8dcd2-24a8-426e-b74e-d90e12c7402c admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.826368 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c07139-3e7e-4f30-96c6-a82231977a20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.827226 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c07139-3e7e-4f30-96c6-a82231977a20 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.827614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c07139-3e7e-4f30-96c6-a82231977a20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.828001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c07139-3e7e-4f30-96c6-a82231977a20 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.828916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c07139-3e7e-4f30-96c6-a82231977a20 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.831037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c07139-3e7e-4f30-96c6-a82231977a20 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.848720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1056/2120] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:41 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:41.849797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c07139-3e7e-4f30-96c6-a82231977a20 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.862337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c07139-3e7e-4f30-96c6-a82231977a20 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.870846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0fdab0-4913-44fa-896e-f7eb1f879f9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.871898 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0fdab0-4913-44fa-896e-f7eb1f879f9b admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.872289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0fdab0-4913-44fa-896e-f7eb1f879f9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.872947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0fdab0-4913-44fa-896e-f7eb1f879f9b admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.873897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0fdab0-4913-44fa-896e-f7eb1f879f9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.875676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b0fdab0-4913-44fa-896e-f7eb1f879f9b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.889182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1057/2121] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:41 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:41.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0fdab0-4913-44fa-896e-f7eb1f879f9b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:41.904664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0fdab0-4913-44fa-896e-f7eb1f879f9b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:41.909817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f455860-7eb1-4f5d-9faa-ae514d0c4d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:41.911007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f455860-7eb1-4f5d-9faa-ae514d0c4d25 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:41.911551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f455860-7eb1-4f5d-9faa-ae514d0c4d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:41.912884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f455860-7eb1-4f5d-9faa-ae514d0c4d25 admin admin] PATH_INFO: `/v3/projects/a4d6211e3bb74ce5ae41b8baa18063fa/users/22dc717ad084423aae080d62bdb078b9/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:41.954664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f455860-7eb1-4f5d-9faa-ae514d0c4d25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4d6211e3bb74ce5ae41b8baa18063fa, user_id=22dc717ad084423aae080d62bdb078b9, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:41.961464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f455860-7eb1-4f5d-9faa-ae514d0c4d25 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:41.976126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ec25481febb4747bcb07175347ed0bb', 'request_id': 'req-5bad824a-ea59-4283-95fb-0f0e5271e678'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:41.995179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ec25481febb4747bcb07175347ed0bb', 'request_id': 'req-5bad824a-ea59-4283-95fb-0f0e5271e678'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:42.003110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ec25481febb4747bcb07175347ed0bb', 'request_id': 'req-5bad824a-ea59-4283-95fb-0f0e5271e678'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:42.067033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1058/2122] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:41 2022] PUT /identity/v3/projects/a4d6211e3bb74ce5ae41b8baa18063fa/users/22dc717ad084423aae080d62bdb078b9/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:42.074073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f455860-7eb1-4f5d-9faa-ae514d0c4d25 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:42.133484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] The token cache is being invalidate because project 8ec25481febb4747bcb07175347ed0bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:42.133890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:42.134632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5bad824a-ea59-4283-95fb-0f0e5271e678 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:42.138500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1065/2123] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:41 2022] DELETE /identity/v3/projects/8ec25481febb4747bcb07175347ed0bb => generated 0 bytes in 487 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:42.140814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab633995-1800-4547-8aa4-299b6c3613cd None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.140814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab633995-1800-4547-8aa4-299b6c3613cd None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.141300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab633995-1800-4547-8aa4-299b6c3613cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.178814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ab633995-1800-4547-8aa4-299b6c3613cd None None] MFA Rules not processed for user `22dc717ad084423aae080d62bdb078b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:42.180061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ab633995-1800-4547-8aa4-299b6c3613cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:42.200751 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f455860-7eb1-4f5d-9faa-ae514d0c4d25 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:42.218865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2553f000-3347-4edb-8129-c689f1fecebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:42.219793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2553f000-3347-4edb-8129-c689f1fecebe admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.220081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2553f000-3347-4edb-8129-c689f1fecebe admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.220370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2553f000-3347-4edb-8129-c689f1fecebe admin admin] PATH_INFO: `/v3/projects/16c0453a7daf4a6d9eb341d4750adfd7/users/14bc57808662457ca339bc86b17d0bda/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.245961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2553f000-3347-4edb-8129-c689f1fecebe admin admin] RBAC: Authorizing `identity:create_grant(project_id=16c0453a7daf4a6d9eb341d4750adfd7, user_id=14bc57808662457ca339bc86b17d0bda, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:42.250017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2553f000-3347-4edb-8129-c689f1fecebe admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:42.292878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1066/2124] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:42.298780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f264e-678a-4372-96e0-514c1908b8f3 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.299067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f264e-678a-4372-96e0-514c1908b8f3 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.299345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f264e-678a-4372-96e0-514c1908b8f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.321026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c15f264e-678a-4372-96e0-514c1908b8f3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:42.329469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c15f264e-678a-4372-96e0-514c1908b8f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:42.342076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1067/2125] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:42.343122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15f264e-678a-4372-96e0-514c1908b8f3 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:42.344928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c15f264e-678a-4372-96e0-514c1908b8f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:42.359480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15f264e-678a-4372-96e0-514c1908b8f3 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:42.375729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1059/2126] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:42 2022] PUT /identity/v3/projects/16c0453a7daf4a6d9eb341d4750adfd7/users/14bc57808662457ca339bc86b17d0bda/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:42.377044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553c458e-c46f-4be7-951d-97d66a97d1f9 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.377434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553c458e-c46f-4be7-951d-97d66a97d1f9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.377761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553c458e-c46f-4be7-951d-97d66a97d1f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.405051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-553c458e-c46f-4be7-951d-97d66a97d1f9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:42.406282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-553c458e-c46f-4be7-951d-97d66a97d1f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:42.450317 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13505924-16ef-4fe4-9b75-80a8e78e8d23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:42.451131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13505924-16ef-4fe4-9b75-80a8e78e8d23 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.451293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13505924-16ef-4fe4-9b75-80a8e78e8d23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.451563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13505924-16ef-4fe4-9b75-80a8e78e8d23 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.452264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13505924-16ef-4fe4-9b75-80a8e78e8d23 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:42.457182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13505924-16ef-4fe4-9b75-80a8e78e8d23 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:42.478563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1060/2127] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:42.480685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad9cf2c-058a-48da-8e07-2aedc0c64b15 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.481239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad9cf2c-058a-48da-8e07-2aedc0c64b15 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.481530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad9cf2c-058a-48da-8e07-2aedc0c64b15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.513527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-cad9cf2c-058a-48da-8e07-2aedc0c64b15 None None] MFA Rules not processed for user `14bc57808662457ca339bc86b17d0bda`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:42.514958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-cad9cf2c-058a-48da-8e07-2aedc0c64b15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:42.557907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1068/2128] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/projects => generated 379 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:42.557907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13505924-16ef-4fe4-9b75-80a8e78e8d23 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:42.590323 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1061/2129] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:42.591473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad9cf2c-058a-48da-8e07-2aedc0c64b15 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:42.611646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad9cf2c-058a-48da-8e07-2aedc0c64b15 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:42.660793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13505924-16ef-4fe4-9b75-80a8e78e8d23 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:42.667926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:42.669025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.669490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.670007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.672129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:42.674329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:42.675813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:42.699204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1069/2130] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:06:42.703713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:42.704507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:42.706793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04417323-8441-44b9-b3f5-47e067e03372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:42.708168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04417323-8441-44b9-b3f5-47e067e03372 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.708692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04417323-8441-44b9-b3f5-47e067e03372 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.709156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04417323-8441-44b9-b3f5-47e067e03372 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.709973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04417323-8441-44b9-b3f5-47e067e03372 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:42.712207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04417323-8441-44b9-b3f5-47e067e03372 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:42.720457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b236fd7-d541-4e09-b2b7-ca47095aa0ae admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:42.726250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8c46b6-2270-4c59-a13b-9803ef6e9b93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:42.726942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8c46b6-2270-4c59-a13b-9803ef6e9b93 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.727216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8c46b6-2270-4c59-a13b-9803ef6e9b93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.727501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8c46b6-2270-4c59-a13b-9803ef6e9b93 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.728191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8c46b6-2270-4c59-a13b-9803ef6e9b93 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:42.730220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8c46b6-2270-4c59-a13b-9803ef6e9b93 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:42.812600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1062/2131] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/users => generated 296 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:42.812600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04417323-8441-44b9-b3f5-47e067e03372 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:42.820971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04417323-8441-44b9-b3f5-47e067e03372 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:42.827027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eae81bb-6b5d-428f-af07-450c49472f46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:42.827666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae81bb-6b5d-428f-af07-450c49472f46 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.827955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae81bb-6b5d-428f-af07-450c49472f46 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.828229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae81bb-6b5d-428f-af07-450c49472f46 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.828946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eae81bb-6b5d-428f-af07-450c49472f46 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:42.831003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eae81bb-6b5d-428f-af07-450c49472f46 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:42.844344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1070/2132] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/projects => generated 401 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:42.844344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8c46b6-2270-4c59-a13b-9803ef6e9b93 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:42.917487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8c46b6-2270-4c59-a13b-9803ef6e9b93 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:42.923053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1277736b-3f4f-4b72-b7f5-636c0c3f69b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:42.924596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1277736b-3f4f-4b72-b7f5-636c0c3f69b1 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:42.924923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1277736b-3f4f-4b72-b7f5-636c0c3f69b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:42.925209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1277736b-3f4f-4b72-b7f5-636c0c3f69b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:42.926089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1277736b-3f4f-4b72-b7f5-636c0c3f69b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:42.927859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1277736b-3f4f-4b72-b7f5-636c0c3f69b1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:42.943155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1063/2133] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/projects => generated 383 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:42.947344 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eae81bb-6b5d-428f-af07-450c49472f46 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:42.985526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1071/2134] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:42 2022] GET /identity/v3/roles => generated 1420 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:42.986356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1277736b-3f4f-4b72-b7f5-636c0c3f69b1 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.023545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1277736b-3f4f-4b72-b7f5-636c0c3f69b1 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.078867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eae81bb-6b5d-428f-af07-450c49472f46 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.084230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831d8afd-82ce-4dc4-ad55-eac04877b0de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.085255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831d8afd-82ce-4dc4-ad55-eac04877b0de admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.085779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831d8afd-82ce-4dc4-ad55-eac04877b0de admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.086889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831d8afd-82ce-4dc4-ad55-eac04877b0de admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.087817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831d8afd-82ce-4dc4-ad55-eac04877b0de admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.090078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831d8afd-82ce-4dc4-ad55-eac04877b0de admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.101187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6a3f65-9750-4f64-accb-7c2b500eb7ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.101873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6a3f65-9750-4f64-accb-7c2b500eb7ac admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.102245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6a3f65-9750-4f64-accb-7c2b500eb7ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.102655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db6a3f65-9750-4f64-accb-7c2b500eb7ac admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.103437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6a3f65-9750-4f64-accb-7c2b500eb7ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.105769 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db6a3f65-9750-4f64-accb-7c2b500eb7ac admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.223464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1072/2135] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/users => generated 292 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:43.223464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6a3f65-9750-4f64-accb-7c2b500eb7ac admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.226902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1064/2136] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:42 2022] POST /identity/v3/users => generated 307 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:43.227855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831d8afd-82ce-4dc4-ad55-eac04877b0de admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.240217 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831d8afd-82ce-4dc4-ad55-eac04877b0de admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.255602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db6a3f65-9750-4f64-accb-7c2b500eb7ac admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.269476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20307a0-017f-4585-860e-18b4a22e2147 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.269655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebfa981-4cc7-4b64-86c8-9fe95ba31731 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.270499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20307a0-017f-4585-860e-18b4a22e2147 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.270738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebfa981-4cc7-4b64-86c8-9fe95ba31731 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.270952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20307a0-017f-4585-860e-18b4a22e2147 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.271064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebfa981-4cc7-4b64-86c8-9fe95ba31731 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.271240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cebfa981-4cc7-4b64-86c8-9fe95ba31731 admin admin] PATH_INFO: `/v3/projects/11515213701a462e81199fadb7b3766f/users/3581b3a93f35425aa961115edc9b8c2b/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.271336 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20307a0-017f-4585-860e-18b4a22e2147 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.271984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20307a0-017f-4585-860e-18b4a22e2147 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.273657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20307a0-017f-4585-860e-18b4a22e2147 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.291288 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1065/2137] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:43 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:43.292851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20307a0-017f-4585-860e-18b4a22e2147 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.293840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebfa981-4cc7-4b64-86c8-9fe95ba31731 admin admin] RBAC: Authorizing `identity:create_grant(project_id=11515213701a462e81199fadb7b3766f, user_id=3581b3a93f35425aa961115edc9b8c2b, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.299952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cebfa981-4cc7-4b64-86c8-9fe95ba31731 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.304936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20307a0-017f-4585-860e-18b4a22e2147 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.310816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363015fe-c036-410a-8a0c-970059f3c5db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.311635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363015fe-c036-410a-8a0c-970059f3c5db admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.311787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363015fe-c036-410a-8a0c-970059f3c5db admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.312055 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-363015fe-c036-410a-8a0c-970059f3c5db admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.312818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363015fe-c036-410a-8a0c-970059f3c5db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.314443 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-363015fe-c036-410a-8a0c-970059f3c5db admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.338155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1066/2138] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:43 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:43.341866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363015fe-c036-410a-8a0c-970059f3c5db admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.353427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-363015fe-c036-410a-8a0c-970059f3c5db admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.359022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b6322f-4ece-496c-bfa7-fd2c561cfd34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.359689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b6322f-4ece-496c-bfa7-fd2c561cfd34 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.360392 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b6322f-4ece-496c-bfa7-fd2c561cfd34 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.360680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b6322f-4ece-496c-bfa7-fd2c561cfd34 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.361466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b6322f-4ece-496c-bfa7-fd2c561cfd34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.363091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9b6322f-4ece-496c-bfa7-fd2c561cfd34 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.373535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1067/2139] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:43 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:43.374328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b6322f-4ece-496c-bfa7-fd2c561cfd34 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.387332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b6322f-4ece-496c-bfa7-fd2c561cfd34 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.392345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef3f8bf-a809-49f5-acda-10f4d9eaa2f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.393255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef3f8bf-a809-49f5-acda-10f4d9eaa2f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.393546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef3f8bf-a809-49f5-acda-10f4d9eaa2f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.393831 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef3f8bf-a809-49f5-acda-10f4d9eaa2f4 admin admin] PATH_INFO: `/v3/projects/cfa59aec48284592835d9fc2c4af993a/users/89e4d52972374ec68be0b91dcdc265f2/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.409302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef3f8bf-a809-49f5-acda-10f4d9eaa2f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfa59aec48284592835d9fc2c4af993a, user_id=89e4d52972374ec68be0b91dcdc265f2, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.414138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ef3f8bf-a809-49f5-acda-10f4d9eaa2f4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.437763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1073/2140] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:43 2022] PUT /identity/v3/projects/11515213701a462e81199fadb7b3766f/users/3581b3a93f35425aa961115edc9b8c2b/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:43.438632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebfa981-4cc7-4b64-86c8-9fe95ba31731 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.498353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cebfa981-4cc7-4b64-86c8-9fe95ba31731 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.500667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1068/2141] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:43 2022] PUT /identity/v3/projects/cfa59aec48284592835d9fc2c4af993a/users/89e4d52972374ec68be0b91dcdc265f2/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:43.504008 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef3f8bf-a809-49f5-acda-10f4d9eaa2f4 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.526053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:DELETE=2,keystone:SELECT=95,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:43.556479 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a882e2a4-af00-4e52-be23-2eb0bda9bd1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.557400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a882e2a4-af00-4e52-be23-2eb0bda9bd1e admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.557707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a882e2a4-af00-4e52-be23-2eb0bda9bd1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.557985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a882e2a4-af00-4e52-be23-2eb0bda9bd1e admin admin] PATH_INFO: `/v3/projects/ff2616337ae440a89047fd9a50218387/users/849e36f8653d4d1f9a7fb1df8fb913a9/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.565095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef3f8bf-a809-49f5-acda-10f4d9eaa2f4 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.570014 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f448bf-6519-4c00-86c1-9e74f27584ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.570715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f448bf-6519-4c00-86c1-9e74f27584ec admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.571118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f448bf-6519-4c00-86c1-9e74f27584ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.571410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f448bf-6519-4c00-86c1-9e74f27584ec admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.572149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f448bf-6519-4c00-86c1-9e74f27584ec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.572256 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a882e2a4-af00-4e52-be23-2eb0bda9bd1e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ff2616337ae440a89047fd9a50218387, user_id=849e36f8653d4d1f9a7fb1df8fb913a9, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.574362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f448bf-6519-4c00-86c1-9e74f27584ec admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.575389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a882e2a4-af00-4e52-be23-2eb0bda9bd1e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.584296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1069/2142] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:43 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:43.585142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f448bf-6519-4c00-86c1-9e74f27584ec admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.595433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f448bf-6519-4c00-86c1-9e74f27584ec admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.599993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c0c9bb-3420-47ca-90ed-9bac40878514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.600622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c0c9bb-3420-47ca-90ed-9bac40878514 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.600936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c0c9bb-3420-47ca-90ed-9bac40878514 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.601212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c0c9bb-3420-47ca-90ed-9bac40878514 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.601960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c0c9bb-3420-47ca-90ed-9bac40878514 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.604091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:43.605716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c0c9bb-3420-47ca-90ed-9bac40878514 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.618841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1070/2143] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:43.618841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c0c9bb-3420-47ca-90ed-9bac40878514 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.626257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c0c9bb-3420-47ca-90ed-9bac40878514 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.631101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229942c2-7927-4e3a-b590-ae49299e3c41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.631983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229942c2-7927-4e3a-b590-ae49299e3c41 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.635413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229942c2-7927-4e3a-b590-ae49299e3c41 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.635413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-229942c2-7927-4e3a-b590-ae49299e3c41 admin admin] PATH_INFO: `/v3/projects/11515213701a462e81199fadb7b3766f/users/3581b3a93f35425aa961115edc9b8c2b/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.635413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229942c2-7927-4e3a-b590-ae49299e3c41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=11515213701a462e81199fadb7b3766f, user_id=3581b3a93f35425aa961115edc9b8c2b, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.638071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-229942c2-7927-4e3a-b590-ae49299e3c41 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.649954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1071/2144] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:43 2022] PUT /identity/v3/projects/11515213701a462e81199fadb7b3766f/users/3581b3a93f35425aa961115edc9b8c2b/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:43.650747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229942c2-7927-4e3a-b590-ae49299e3c41 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.661281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1074/2145] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:43 2022] PUT /identity/v3/projects/ff2616337ae440a89047fd9a50218387/users/849e36f8653d4d1f9a7fb1df8fb913a9/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:43.662393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22b079b-b5d4-487d-b3f5-a466eef8b5e6 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.662685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22b079b-b5d4-487d-b3f5-a466eef8b5e6 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.662965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22b079b-b5d4-487d-b3f5-a466eef8b5e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.692571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c22b079b-b5d4-487d-b3f5-a466eef8b5e6 None None] MFA Rules not processed for user `3581b3a93f35425aa961115edc9b8c2b`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:43.694016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c22b079b-b5d4-487d-b3f5-a466eef8b5e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:43.727305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-229942c2-7927-4e3a-b590-ae49299e3c41 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.773003 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1075/2146] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:43 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:43.776355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01c470f-5c83-45fc-a2bb-0e99c62d6fff None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.776651 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01c470f-5c83-45fc-a2bb-0e99c62d6fff None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.776960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01c470f-5c83-45fc-a2bb-0e99c62d6fff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.802396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5265fc63-7f0e-42d2-8a8f-be45cfef5d6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.803330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5265fc63-7f0e-42d2-8a8f-be45cfef5d6d admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.803617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5265fc63-7f0e-42d2-8a8f-be45cfef5d6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.803772 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a01c470f-5c83-45fc-a2bb-0e99c62d6fff None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:43.803905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5265fc63-7f0e-42d2-8a8f-be45cfef5d6d admin admin] PATH_INFO: `/v3/projects/cfa59aec48284592835d9fc2c4af993a/users/89e4d52972374ec68be0b91dcdc265f2/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.804999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a01c470f-5c83-45fc-a2bb-0e99c62d6fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:43.807249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5265fc63-7f0e-42d2-8a8f-be45cfef5d6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=cfa59aec48284592835d9fc2c4af993a, user_id=89e4d52972374ec68be0b91dcdc265f2, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.810314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5265fc63-7f0e-42d2-8a8f-be45cfef5d6d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.814380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1076/2147] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:43.815331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01c470f-5c83-45fc-a2bb-0e99c62d6fff None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.818026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a01c470f-5c83-45fc-a2bb-0e99c62d6fff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:43.821458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1072/2148] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:43 2022] PUT /identity/v3/projects/cfa59aec48284592835d9fc2c4af993a/users/89e4d52972374ec68be0b91dcdc265f2/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:43.830922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5265fc63-7f0e-42d2-8a8f-be45cfef5d6d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:43.831870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5265fc63-7f0e-42d2-8a8f-be45cfef5d6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:43.880090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01c470f-5c83-45fc-a2bb-0e99c62d6fff None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.891558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5265fc63-7f0e-42d2-8a8f-be45cfef5d6d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:43.896332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a686c559-a024-42a4-aa27-2e48f5f16140 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.897026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a686c559-a024-42a4-aa27-2e48f5f16140 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.897317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a686c559-a024-42a4-aa27-2e48f5f16140 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.897600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a686c559-a024-42a4-aa27-2e48f5f16140 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.934153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86423b76-27bf-42be-b9ad-cbdf52185eb9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:43.935628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86423b76-27bf-42be-b9ad-cbdf52185eb9 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.935628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86423b76-27bf-42be-b9ad-cbdf52185eb9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.936442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86423b76-27bf-42be-b9ad-cbdf52185eb9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:43.968423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a686c559-a024-42a4-aa27-2e48f5f16140 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:43.970069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a686c559-a024-42a4-aa27-2e48f5f16140 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:43.982570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1073/2149] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:43.983897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47b4dde-a664-4fd6-a688-9c489051a5ac None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:43.984190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47b4dde-a664-4fd6-a688-9c489051a5ac None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:43.984476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47b4dde-a664-4fd6-a688-9c489051a5ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:44.020412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-f47b4dde-a664-4fd6-a688-9c489051a5ac None None] MFA Rules not processed for user `89e4d52972374ec68be0b91dcdc265f2`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:44.021743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f47b4dde-a664-4fd6-a688-9c489051a5ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:44.036985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86423b76-27bf-42be-b9ad-cbdf52185eb9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:44.039425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86423b76-27bf-42be-b9ad-cbdf52185eb9 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:44.050236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1077/2150] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:43 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:44.104786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1074/2151] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:43 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:44.113062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86423b76-27bf-42be-b9ad-cbdf52185eb9 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:44.114540 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-86423b76-27bf-42be-b9ad-cbdf52185eb9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:44.128053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86423b76-27bf-42be-b9ad-cbdf52185eb9 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:44.131942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce440d6-476e-40da-b9ed-86d2ab4dd36b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:44.136654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce440d6-476e-40da-b9ed-86d2ab4dd36b service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:44.136654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce440d6-476e-40da-b9ed-86d2ab4dd36b service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:44.136654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce440d6-476e-40da-b9ed-86d2ab4dd36b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:44.147179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce440d6-476e-40da-b9ed-86d2ab4dd36b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:44.149180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce440d6-476e-40da-b9ed-86d2ab4dd36b service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:44.159484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1078/2152] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:44 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:44.197687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47b4dde-a664-4fd6-a688-9c489051a5ac None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:44.207696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f47b4dde-a664-4fd6-a688-9c489051a5ac None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:44.211557 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6171e7-1737-49e7-bcf7-c0f4d13214c6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:44.212206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6171e7-1737-49e7-bcf7-c0f4d13214c6 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:44.212492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6171e7-1737-49e7-bcf7-c0f4d13214c6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:44.213382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6171e7-1737-49e7-bcf7-c0f4d13214c6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:44.215486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2e6171e7-1737-49e7-bcf7-c0f4d13214c6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:44.303278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6171e7-1737-49e7-bcf7-c0f4d13214c6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:44.306223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6171e7-1737-49e7-bcf7-c0f4d13214c6 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:44.342497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1075/2153] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:44 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:44.413677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce440d6-476e-40da-b9ed-86d2ab4dd36b service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:44.423728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce440d6-476e-40da-b9ed-86d2ab4dd36b service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:44.427598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ece80a0-0aa8-44e5-ab27-728ceddc8410 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:44.428469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ece80a0-0aa8-44e5-ab27-728ceddc8410 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:44.428774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ece80a0-0aa8-44e5-ab27-728ceddc8410 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:44.429070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ece80a0-0aa8-44e5-ab27-728ceddc8410 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:44.489312 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ece80a0-0aa8-44e5-ab27-728ceddc8410 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:44.490782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ece80a0-0aa8-44e5-ab27-728ceddc8410 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:44.503561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1079/2154] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:44 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:45.567186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68762e6-6653-4f0b-9194-7c3949046f8d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:45.567637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68762e6-6653-4f0b-9194-7c3949046f8d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:45.567809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68762e6-6653-4f0b-9194-7c3949046f8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:45.599221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-b68762e6-6653-4f0b-9194-7c3949046f8d None None] MFA Rules not processed for user `849e36f8653d4d1f9a7fb1df8fb913a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:45.601551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b68762e6-6653-4f0b-9194-7c3949046f8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:45.673266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1076/2155] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:45 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:45.684233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ece80a0-0aa8-44e5-ab27-728ceddc8410 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:45.694418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ece80a0-0aa8-44e5-ab27-728ceddc8410 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:45.704268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae962c0-a0b1-42ef-9f7e-6cc25a576232 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:45.705218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae962c0-a0b1-42ef-9f7e-6cc25a576232 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:45.705649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae962c0-a0b1-42ef-9f7e-6cc25a576232 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:45.706486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae962c0-a0b1-42ef-9f7e-6cc25a576232 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:45.718736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae962c0-a0b1-42ef-9f7e-6cc25a576232 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:45.720909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae962c0-a0b1-42ef-9f7e-6cc25a576232 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:45.728517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:45.735313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1080/2156] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:45 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:45.819069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:47.519041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb55712-342f-4e00-83b5-fc0b58d87b0a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:47.519822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb55712-342f-4e00-83b5-fc0b58d87b0a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:47.519822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb55712-342f-4e00-83b5-fc0b58d87b0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:47.539096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1eb55712-342f-4e00-83b5-fc0b58d87b0a None None] MFA Rules not processed for user `849e36f8653d4d1f9a7fb1df8fb913a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:47.540325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1eb55712-342f-4e00-83b5-fc0b58d87b0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:47.549536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1077/2157] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:47 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:47.826498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:47.870852 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adb731-d90c-414c-92d7-cc752d5eff5c None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:47.871152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adb731-d90c-414c-92d7-cc752d5eff5c None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:47.871437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1adb731-d90c-414c-92d7-cc752d5eff5c None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:47.873004 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1081/2158] 10.222.0.25 () {54 vars in 912 bytes} [Fri Jun 10 13:06:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:06:47.877998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:47.881135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05612ee4-ff3b-4b0f-9ba5-415124c0664d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:47.881786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05612ee4-ff3b-4b0f-9ba5-415124c0664d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:47.882294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05612ee4-ff3b-4b0f-9ba5-415124c0664d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:47.902930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-05612ee4-ff3b-4b0f-9ba5-415124c0664d None None] MFA Rules not processed for user `d6794257c90e40bca2c8efd31b0c791a`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:47.904152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-05612ee4-ff3b-4b0f-9ba5-415124c0664d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:47.912738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1078/2159] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:06:47 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:47.924325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1adb731-d90c-414c-92d7-cc752d5eff5c None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:47.936250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1adb731-d90c-414c-92d7-cc752d5eff5c None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:47.939798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337c20be-c262-49af-88a4-5b3449a6c824 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:47.940426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c20be-c262-49af-88a4-5b3449a6c824 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:47.940706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c20be-c262-49af-88a4-5b3449a6c824 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:47.941017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337c20be-c262-49af-88a4-5b3449a6c824 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:47.948122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337c20be-c262-49af-88a4-5b3449a6c824 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:47.949575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337c20be-c262-49af-88a4-5b3449a6c824 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:47.958298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1082/2160] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:47 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:48.928995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05612ee4-ff3b-4b0f-9ba5-415124c0664d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:48.941386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05612ee4-ff3b-4b0f-9ba5-415124c0664d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:48.946016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:48.946887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:48.947560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:48.947842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] PATH_INFO: `/v3/users/dccc9b250645451c94b15aae40beff0f` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:48.950431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dccc9b250645451c94b15aae40beff0f)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:48.952372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:49.310215 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dccc9b250645451c94b15aae40beff0f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:49.315699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dccc9b250645451c94b15aae40beff0f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:49.315699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dccc9b250645451c94b15aae40beff0f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:49.330691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dccc9b250645451c94b15aae40beff0f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:49.397573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dccc9b250645451c94b15aae40beff0f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:49.435306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1079/2161] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:48 2022] DELETE /identity/v3/users/dccc9b250645451c94b15aae40beff0f => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:49.437096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed87ae-877f-4e00-aef4-1e24f304ce56 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:49.437470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed87ae-877f-4e00-aef4-1e24f304ce56 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:49.437623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ed87ae-877f-4e00-aef4-1e24f304ce56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:49.456431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-15ed87ae-877f-4e00-aef4-1e24f304ce56 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:49.457689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-15ed87ae-877f-4e00-aef4-1e24f304ce56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:49.574082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1083/2162] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:49.583824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:49.586958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:49.652637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18384870-248d-4935-b5b4-d067b60071b4 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:49.656372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad65571-2dc2-4447-b75b-5446d44e4203 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:49.657372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad65571-2dc2-4447-b75b-5446d44e4203 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:49.657660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad65571-2dc2-4447-b75b-5446d44e4203 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:49.657940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad65571-2dc2-4447-b75b-5446d44e4203 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:49.668588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad65571-2dc2-4447-b75b-5446d44e4203 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:49.670894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad65571-2dc2-4447-b75b-5446d44e4203 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:49.684971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1080/2163] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:06:50.125303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed87ae-877f-4e00-aef4-1e24f304ce56 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:50.129239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-15ed87ae-877f-4e00-aef4-1e24f304ce56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:50.139803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ed87ae-877f-4e00-aef4-1e24f304ce56 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:50.145013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:50.146034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:50.146322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:50.146608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] PATH_INFO: `/v3/projects/efa7b57104734551afbc94c410a21dad` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:50.147864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] RBAC: Authorizing `identity:delete_project(project_id=efa7b57104734551afbc94c410a21dad)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:50.150096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:50.180383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=9,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:50.268403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'efa7b57104734551afbc94c410a21dad', 'request_id': 'req-db82c83c-a7d1-4a80-87cb-a070b5a18822'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.275326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'efa7b57104734551afbc94c410a21dad', 'request_id': 'req-db82c83c-a7d1-4a80-87cb-a070b5a18822'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.281297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'efa7b57104734551afbc94c410a21dad', 'request_id': 'req-db82c83c-a7d1-4a80-87cb-a070b5a18822'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.387506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] The token cache is being invalidate because project efa7b57104734551afbc94c410a21dad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:50.387717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.388356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.390245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1084/2164] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:50 2022] DELETE /identity/v3/projects/efa7b57104734551afbc94c410a21dad => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:50.394797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad65571-2dc2-4447-b75b-5446d44e4203 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:50.405628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad65571-2dc2-4447-b75b-5446d44e4203 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:50.464914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:50.465786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:50.466072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:50.466376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] PATH_INFO: `/v3/users/22dc717ad084423aae080d62bdb078b9` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:50.467605 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] RBAC: Authorizing `identity:delete_user(user_id=22dc717ad084423aae080d62bdb078b9)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:50.469582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:50.711233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '22dc717ad084423aae080d62bdb078b9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.712234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '22dc717ad084423aae080d62bdb078b9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.713082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '22dc717ad084423aae080d62bdb078b9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.721996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '22dc717ad084423aae080d62bdb078b9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.843948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '22dc717ad084423aae080d62bdb078b9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:50.863590 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1081/2165] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:50 2022] DELETE /identity/v3/users/22dc717ad084423aae080d62bdb078b9 => generated 0 bytes in 469 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:51.364039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:51.365541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:51.379314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:51.385133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:51.389021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db82c83c-a7d1-4a80-87cb-a070b5a18822 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:51.449366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44569d2f-a444-4f17-9871-077525ff3409 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:51.453467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:51.454452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:51.454768 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:51.468618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:51.468946 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] PATH_INFO: `/v3/projects/a4d6211e3bb74ce5ae41b8baa18063fa` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:51.469052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:51.469132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4d6211e3bb74ce5ae41b8baa18063fa)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:51.469203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:51.469203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] PATH_INFO: `/v3/users/14bc57808662457ca339bc86b17d0bda` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:51.469203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] RBAC: Authorizing `identity:delete_user(user_id=14bc57808662457ca339bc86b17d0bda)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:51.469478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:51.469571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:51.691829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4d6211e3bb74ce5ae41b8baa18063fa', 'request_id': 'req-c76f57da-7f40-4c62-a401-9483bd279475'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.692009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '14bc57808662457ca339bc86b17d0bda'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.692688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '14bc57808662457ca339bc86b17d0bda'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.693516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '14bc57808662457ca339bc86b17d0bda'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.698324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4d6211e3bb74ce5ae41b8baa18063fa', 'request_id': 'req-c76f57da-7f40-4c62-a401-9483bd279475'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.700994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '14bc57808662457ca339bc86b17d0bda'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.705875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4d6211e3bb74ce5ae41b8baa18063fa', 'request_id': 'req-c76f57da-7f40-4c62-a401-9483bd279475'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.803381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-78d16230-776e-4512-b9de-66fbfaa71e61 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '14bc57808662457ca339bc86b17d0bda'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.822677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1082/2166] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:51 2022] DELETE /identity/v3/users/14bc57808662457ca339bc86b17d0bda => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:51.826988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8277242-4eab-45c6-a108-9b2249e06bb2 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:51.827276 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8277242-4eab-45c6-a108-9b2249e06bb2 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:51.827553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8277242-4eab-45c6-a108-9b2249e06bb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:51.851134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a8277242-4eab-45c6-a108-9b2249e06bb2 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:51.852365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a8277242-4eab-45c6-a108-9b2249e06bb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:51.904707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] The token cache is being invalidate because project a4d6211e3bb74ce5ae41b8baa18063fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:51.909743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.909743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:51.909743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1085/2167] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:51 2022] DELETE /identity/v3/projects/a4d6211e3bb74ce5ae41b8baa18063fa => generated 0 bytes in 544 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:51.912937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:51.914911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:51.935785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1083/2168] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:51.943088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8277242-4eab-45c6-a108-9b2249e06bb2 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:51.947898 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a8277242-4eab-45c6-a108-9b2249e06bb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:52.010016 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76f57da-7f40-4c62-a401-9483bd279475 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:52.015988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:52.016892 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:52.017348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:52.017687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] PATH_INFO: `/v3/users/3581b3a93f35425aa961115edc9b8c2b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:52.024265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:52.024409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=3581b3a93f35425aa961115edc9b8c2b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:52.024409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:52.036433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8277242-4eab-45c6-a108-9b2249e06bb2 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:52.040344 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7e4db1-2d99-420a-9b8a-68ce6592aedc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:52.041029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7e4db1-2d99-420a-9b8a-68ce6592aedc service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:52.047022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7e4db1-2d99-420a-9b8a-68ce6592aedc service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:52.047022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab7e4db1-2d99-420a-9b8a-68ce6592aedc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:52.051392 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7e4db1-2d99-420a-9b8a-68ce6592aedc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:52.052988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab7e4db1-2d99-420a-9b8a-68ce6592aedc service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:52.062709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1084/2169] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:52.268849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3581b3a93f35425aa961115edc9b8c2b'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.269575 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3581b3a93f35425aa961115edc9b8c2b'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.276714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3581b3a93f35425aa961115edc9b8c2b'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.341446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3581b3a93f35425aa961115edc9b8c2b'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.342157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3581b3a93f35425aa961115edc9b8c2b'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.362051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1086/2170] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:51 2022] DELETE /identity/v3/users/3581b3a93f35425aa961115edc9b8c2b => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:52.519836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7e4db1-2d99-420a-9b8a-68ce6592aedc service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:52.521253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ab7e4db1-2d99-420a-9b8a-68ce6592aedc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:52.537694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab7e4db1-2d99-420a-9b8a-68ce6592aedc service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:52.602952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:52.604103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:52.604493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:52.605204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] PATH_INFO: `/v3/projects/16c0453a7daf4a6d9eb341d4750adfd7` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:52.606613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=16c0453a7daf4a6d9eb341d4750adfd7)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:52.608909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:52.728757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '16c0453a7daf4a6d9eb341d4750adfd7', 'request_id': 'req-bba9c452-d1da-4fa9-9411-17657ed61cc7'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.798979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:52.801053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:52.816864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220e0c3e-0f40-45c1-ad89-04ec20dcfd0b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:52.829474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '16c0453a7daf4a6d9eb341d4750adfd7', 'request_id': 'req-bba9c452-d1da-4fa9-9411-17657ed61cc7'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.848305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '16c0453a7daf4a6d9eb341d4750adfd7', 'request_id': 'req-bba9c452-d1da-4fa9-9411-17657ed61cc7'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.860190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] The token cache is being invalidate because project 16c0453a7daf4a6d9eb341d4750adfd7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:52.860482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.861282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:52.862843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1085/2171] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:52 2022] DELETE /identity/v3/projects/16c0453a7daf4a6d9eb341d4750adfd7 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:52.866605 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:52.867922 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:52.884301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:52.885204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:52.885489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:52.887685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] PATH_INFO: `/v3/projects/11515213701a462e81199fadb7b3766f` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:52.887685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] RBAC: Authorizing `identity:delete_project(project_id=11515213701a462e81199fadb7b3766f)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:52.889012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:52.934641 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bba9c452-d1da-4fa9-9411-17657ed61cc7 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:52.939520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:52.940472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:52.940751 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:52.941052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] PATH_INFO: `/v3/users/89e4d52972374ec68be0b91dcdc265f2` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:52.942256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89e4d52972374ec68be0b91dcdc265f2)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:52.950754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:53.057835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '11515213701a462e81199fadb7b3766f', 'request_id': 'req-9c80dbe5-130e-4f24-a857-d191a7d62b6f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.065058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '11515213701a462e81199fadb7b3766f', 'request_id': 'req-9c80dbe5-130e-4f24-a857-d191a7d62b6f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.070768 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '11515213701a462e81199fadb7b3766f', 'request_id': 'req-9c80dbe5-130e-4f24-a857-d191a7d62b6f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.179884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] The token cache is being invalidate because project 11515213701a462e81199fadb7b3766f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:53.180183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.180831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9c80dbe5-130e-4f24-a857-d191a7d62b6f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.182478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1087/2172] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:52 2022] DELETE /identity/v3/projects/11515213701a462e81199fadb7b3766f => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:06:53.188222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6507e605-82c6-4470-bce6-6c0597af27ea None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.188514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6507e605-82c6-4470-bce6-6c0597af27ea None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.188827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6507e605-82c6-4470-bce6-6c0597af27ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.189792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89e4d52972374ec68be0b91dcdc265f2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.190465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89e4d52972374ec68be0b91dcdc265f2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.191095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89e4d52972374ec68be0b91dcdc265f2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.198919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89e4d52972374ec68be0b91dcdc265f2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.207180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6507e605-82c6-4470-bce6-6c0597af27ea None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:53.208415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6507e605-82c6-4470-bce6-6c0597af27ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:53.266337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1088/2173] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:53.274494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a17d39a-2be2-4448-9f2a-b0013113f710 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.274856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a17d39a-2be2-4448-9f2a-b0013113f710 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.275130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a17d39a-2be2-4448-9f2a-b0013113f710 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.293890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-0a17d39a-2be2-4448-9f2a-b0013113f710 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:53.295088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0a17d39a-2be2-4448-9f2a-b0013113f710 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:53.303075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1089/2174] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:53.306893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a17d39a-2be2-4448-9f2a-b0013113f710 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:53.315656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a17d39a-2be2-4448-9f2a-b0013113f710 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:53.320325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb8364-12c5-4620-a503-1495cbaa0ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:53.321082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb8364-12c5-4620-a503-1495cbaa0ac9 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.321375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb8364-12c5-4620-a503-1495cbaa0ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.321659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb8364-12c5-4620-a503-1495cbaa0ac9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.322377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb8364-12c5-4620-a503-1495cbaa0ac9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:53.322688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb8364-12c5-4620-a503-1495cbaa0ac9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:53.324332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb8364-12c5-4620-a503-1495cbaa0ac9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:53.328869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89e4d52972374ec68be0b91dcdc265f2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.337080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1090/2175] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:06:53.342043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb8364-12c5-4620-a503-1495cbaa0ac9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:53.352543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb8364-12c5-4620-a503-1495cbaa0ac9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:53.352699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1086/2176] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:52 2022] DELETE /identity/v3/users/89e4d52972374ec68be0b91dcdc265f2 => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:53.414282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcf107b-35bd-4fe0-be91-fa9d57431af0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:53.414951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcf107b-35bd-4fe0-be91-fa9d57431af0 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.415241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcf107b-35bd-4fe0-be91-fa9d57431af0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.415522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dcf107b-35bd-4fe0-be91-fa9d57431af0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.416215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcf107b-35bd-4fe0-be91-fa9d57431af0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:53.418811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dcf107b-35bd-4fe0-be91-fa9d57431af0 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:53.506529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1091/2177] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:53 2022] POST /identity/v3/projects => generated 383 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:53.510838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:53.521237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18bccba2-c01d-4d78-93a7-a839eca48e05 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:53.588152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb0ed9b-c87d-4d3f-b083-5c61414fae80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:53.588856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb0ed9b-c87d-4d3f-b083-5c61414fae80 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.589139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb0ed9b-c87d-4d3f-b083-5c61414fae80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.589411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb0ed9b-c87d-4d3f-b083-5c61414fae80 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.590109 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb0ed9b-c87d-4d3f-b083-5c61414fae80 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:53.592705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb0ed9b-c87d-4d3f-b083-5c61414fae80 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:53.674740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1087/2178] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:53 2022] POST /identity/v3/users => generated 292 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:53.676134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcf107b-35bd-4fe0-be91-fa9d57431af0 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:53.691664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dcf107b-35bd-4fe0-be91-fa9d57431af0 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:53.698661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0116cdd-ba0c-41a9-a672-f8835ff94aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:53.698661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0116cdd-ba0c-41a9-a672-f8835ff94aa1 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.698661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0116cdd-ba0c-41a9-a672-f8835ff94aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.698661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0116cdd-ba0c-41a9-a672-f8835ff94aa1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.699000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0116cdd-ba0c-41a9-a672-f8835ff94aa1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:53.700166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0116cdd-ba0c-41a9-a672-f8835ff94aa1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:53.711313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1092/2179] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:53 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:53.715302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb0ed9b-c87d-4d3f-b083-5c61414fae80 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:53.726362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb0ed9b-c87d-4d3f-b083-5c61414fae80 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:53.731232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144986da-c300-4133-9ee3-4338c261b26d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:53.731893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144986da-c300-4133-9ee3-4338c261b26d admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.732178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144986da-c300-4133-9ee3-4338c261b26d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.732466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-144986da-c300-4133-9ee3-4338c261b26d admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.733218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144986da-c300-4133-9ee3-4338c261b26d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:53.735071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-144986da-c300-4133-9ee3-4338c261b26d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:53.745071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1088/2180] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:53 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:53.749343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0116cdd-ba0c-41a9-a672-f8835ff94aa1 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:53.759435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0116cdd-ba0c-41a9-a672-f8835ff94aa1 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:53.765452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb8d859-d71d-4d09-bdf8-c8498b79b6db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:53.766356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb8d859-d71d-4d09-bdf8-c8498b79b6db admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.766644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb8d859-d71d-4d09-bdf8-c8498b79b6db admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.766927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb8d859-d71d-4d09-bdf8-c8498b79b6db admin admin] PATH_INFO: `/v3/projects/2607c07d53c74bcf9abf439950317789/users/6162b49b01924cfb834e9c7ab359850f/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.776220 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144986da-c300-4133-9ee3-4338c261b26d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:53.779061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-144986da-c300-4133-9ee3-4338c261b26d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:53.784190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb8d859-d71d-4d09-bdf8-c8498b79b6db admin admin] RBAC: Authorizing `identity:create_grant(project_id=2607c07d53c74bcf9abf439950317789, user_id=6162b49b01924cfb834e9c7ab359850f, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:53.787041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb8d859-d71d-4d09-bdf8-c8498b79b6db admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:53.794874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-144986da-c300-4133-9ee3-4338c261b26d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:53.799811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:53.800684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.801040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.801321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] PATH_INFO: `/v3/projects/cfa59aec48284592835d9fc2c4af993a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.803293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cfa59aec48284592835d9fc2c4af993a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:53.804464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:53.843598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1093/2181] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:53 2022] PUT /identity/v3/projects/2607c07d53c74bcf9abf439950317789/users/6162b49b01924cfb834e9c7ab359850f/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 95 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:53.854298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfa4457-736c-4b65-9b7b-4cb45663bd3b None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:53.854588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfa4457-736c-4b65-9b7b-4cb45663bd3b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:53.854929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfa4457-736c-4b65-9b7b-4cb45663bd3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:53.881058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4cfa4457-736c-4b65-9b7b-4cb45663bd3b None None] MFA Rules not processed for user `6162b49b01924cfb834e9c7ab359850f`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:53.882251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4cfa4457-736c-4b65-9b7b-4cb45663bd3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:53.911521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cfa59aec48284592835d9fc2c4af993a', 'request_id': 'req-731cf3ca-4596-4957-b2e1-9d3f8a330642'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.945184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1094/2182] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:53.956991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfa4457-736c-4b65-9b7b-4cb45663bd3b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:53.958302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4cfa4457-736c-4b65-9b7b-4cb45663bd3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:53.985927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cfa59aec48284592835d9fc2c4af993a', 'request_id': 'req-731cf3ca-4596-4957-b2e1-9d3f8a330642'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:53.995269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cfa59aec48284592835d9fc2c4af993a', 'request_id': 'req-731cf3ca-4596-4957-b2e1-9d3f8a330642'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:54.007077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] The token cache is being invalidate because project cfa59aec48284592835d9fc2c4af993a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:54.007330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:54.007979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-731cf3ca-4596-4957-b2e1-9d3f8a330642 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:54.013614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1089/2183] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:53 2022] DELETE /identity/v3/projects/cfa59aec48284592835d9fc2c4af993a => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:06:54.014092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:INSERT=4,keystone:DELETE=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:54.015062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feb1331-0d87-4550-bb6b-3a6897818c2d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.015356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feb1331-0d87-4550-bb6b-3a6897818c2d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.015640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feb1331-0d87-4550-bb6b-3a6897818c2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.017030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfa4457-736c-4b65-9b7b-4cb45663bd3b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.033597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5feb1331-0d87-4550-bb6b-3a6897818c2d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:54.034866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5feb1331-0d87-4550-bb6b-3a6897818c2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:54.075047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.076623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.076987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.077391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.080091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:54.093338 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:54.101526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1090/2184] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:54.110533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e6cab7-48f1-45d7-aa70-2a7f8750e282 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.110878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e6cab7-48f1-45d7-aa70-2a7f8750e282 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.111167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16e6cab7-48f1-45d7-aa70-2a7f8750e282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.130091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-16e6cab7-48f1-45d7-aa70-2a7f8750e282 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:54.131228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-16e6cab7-48f1-45d7-aa70-2a7f8750e282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:54.139665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.140380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1091/2185] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:54.141153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:54.144682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e6cab7-48f1-45d7-aa70-2a7f8750e282 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:54.150468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1095/2186] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:53 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:06:54.154257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16e6cab7-48f1-45d7-aa70-2a7f8750e282 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.168388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cc21e6-d8c8-470a-b9cf-8ae290d7b42d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.169281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc21e6-d8c8-470a-b9cf-8ae290d7b42d admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.169454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc21e6-d8c8-470a-b9cf-8ae290d7b42d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.179979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cc21e6-d8c8-470a-b9cf-8ae290d7b42d admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.179979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cc21e6-d8c8-470a-b9cf-8ae290d7b42d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:54.179979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cc21e6-d8c8-470a-b9cf-8ae290d7b42d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.179979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19cc21e6-d8c8-470a-b9cf-8ae290d7b42d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:54.180887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:54.180887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:54.197044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1092/2187] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:06:54.197837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cc21e6-d8c8-470a-b9cf-8ae290d7b42d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:54.198736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6627e3-f999-4836-9f92-eea4fc252569 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.206188 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.207320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.207759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.208185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] PATH_INFO: `/v3/users/6162b49b01924cfb834e9c7ab359850f` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.209888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] RBAC: Authorizing `identity:delete_user(user_id=6162b49b01924cfb834e9c7ab359850f)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.210281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cc21e6-d8c8-470a-b9cf-8ae290d7b42d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.220408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:54.220984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979b56b1-2432-4b6b-93f4-fd5cf40be2a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.220984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979b56b1-2432-4b6b-93f4-fd5cf40be2a5 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.220984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979b56b1-2432-4b6b-93f4-fd5cf40be2a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.220984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979b56b1-2432-4b6b-93f4-fd5cf40be2a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.221312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979b56b1-2432-4b6b-93f4-fd5cf40be2a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.221312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-979b56b1-2432-4b6b-93f4-fd5cf40be2a5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:54.317230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1093/2188] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:54 2022] POST /identity/v3/projects => generated 367 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:54.318293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979b56b1-2432-4b6b-93f4-fd5cf40be2a5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:54.381499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-979b56b1-2432-4b6b-93f4-fd5cf40be2a5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.386143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1bb207-cc1e-42a5-a86b-0ef85e6c0eb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.386798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1bb207-cc1e-42a5-a86b-0ef85e6c0eb8 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.387081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1bb207-cc1e-42a5-a86b-0ef85e6c0eb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.387355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1bb207-cc1e-42a5-a86b-0ef85e6c0eb8 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.388042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1bb207-cc1e-42a5-a86b-0ef85e6c0eb8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.390030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1bb207-cc1e-42a5-a86b-0ef85e6c0eb8 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:54.435942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6162b49b01924cfb834e9c7ab359850f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:54.436694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6162b49b01924cfb834e9c7ab359850f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:54.443428 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6162b49b01924cfb834e9c7ab359850f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:54.483517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1094/2189] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:54 2022] POST /identity/v3/users => generated 284 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:54.487664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1bb207-cc1e-42a5-a86b-0ef85e6c0eb8 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:54.489261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-bd1bb207-cc1e-42a5-a86b-0ef85e6c0eb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:54.493886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6162b49b01924cfb834e9c7ab359850f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:54.494507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8a730aa8-b387-4d7c-a0ba-42b98455e18e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6162b49b01924cfb834e9c7ab359850f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:54.502313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1bb207-cc1e-42a5-a86b-0ef85e6c0eb8 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.506990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484e8853-d7ee-4961-9634-80855aa07cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.507728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484e8853-d7ee-4961-9634-80855aa07cc6 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.508038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484e8853-d7ee-4961-9634-80855aa07cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.508351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484e8853-d7ee-4961-9634-80855aa07cc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.509124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484e8853-d7ee-4961-9634-80855aa07cc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.510730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-484e8853-d7ee-4961-9634-80855aa07cc6 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:54.513314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1096/2190] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:06:54 2022] DELETE /identity/v3/users/6162b49b01924cfb834e9c7ab359850f => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:54.517629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e51b144-0e24-42c8-ae0e-740d44f7a01a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.517910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e51b144-0e24-42c8-ae0e-740d44f7a01a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.518178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e51b144-0e24-42c8-ae0e-740d44f7a01a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.521080 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1095/2191] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:54 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:06:54.526471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484e8853-d7ee-4961-9634-80855aa07cc6 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:54.536240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-0e51b144-0e24-42c8-ae0e-740d44f7a01a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:54.536970 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-484e8853-d7ee-4961-9634-80855aa07cc6 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.537467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0e51b144-0e24-42c8-ae0e-740d44f7a01a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:54.599756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd69c851-9053-4528-bbc6-a2ec9e0e286b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.600423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd69c851-9053-4528-bbc6-a2ec9e0e286b admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.600921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd69c851-9053-4528-bbc6-a2ec9e0e286b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.602363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1097/2192] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:54.602481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd69c851-9053-4528-bbc6-a2ec9e0e286b admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.603161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd69c851-9053-4528-bbc6-a2ec9e0e286b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.604819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd69c851-9053-4528-bbc6-a2ec9e0e286b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:54.612582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e51b144-0e24-42c8-ae0e-740d44f7a01a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:54.614137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0e51b144-0e24-42c8-ae0e-740d44f7a01a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:54.626731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1096/2193] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:54 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:06:54.627942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd69c851-9053-4528-bbc6-a2ec9e0e286b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:54.638502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd69c851-9053-4528-bbc6-a2ec9e0e286b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.644219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51aa772-d07d-4a6e-a711-219cbc507879 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.645221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51aa772-d07d-4a6e-a711-219cbc507879 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.650970 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51aa772-d07d-4a6e-a711-219cbc507879 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.650970 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51aa772-d07d-4a6e-a711-219cbc507879 admin admin] PATH_INFO: `/v3/projects/5316f7d5300f4ca6823225ba1db23e96/users/698dc89ee6ef4f41a3b752cc50afddad/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.666097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51aa772-d07d-4a6e-a711-219cbc507879 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5316f7d5300f4ca6823225ba1db23e96, user_id=698dc89ee6ef4f41a3b752cc50afddad, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.668851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51aa772-d07d-4a6e-a711-219cbc507879 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:54.683166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e51b144-0e24-42c8-ae0e-740d44f7a01a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.688470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9b61e8-5fd4-404e-8b56-9a56ea67bae2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.690200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9b61e8-5fd4-404e-8b56-9a56ea67bae2 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.690355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9b61e8-5fd4-404e-8b56-9a56ea67bae2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.690628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9b61e8-5fd4-404e-8b56-9a56ea67bae2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.702269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9b61e8-5fd4-404e-8b56-9a56ea67bae2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.703460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9b61e8-5fd4-404e-8b56-9a56ea67bae2 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:54.715502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1098/2194] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:06:54.718239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1097/2195] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:54 2022] PUT /identity/v3/projects/5316f7d5300f4ca6823225ba1db23e96/users/698dc89ee6ef4f41a3b752cc50afddad/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:54.728201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8372f7-9284-4614-8ba5-e6b42ae17903 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.728561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8372f7-9284-4614-8ba5-e6b42ae17903 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.728989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8372f7-9284-4614-8ba5-e6b42ae17903 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.761115 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-1c8372f7-9284-4614-8ba5-e6b42ae17903 None None] MFA Rules not processed for user `698dc89ee6ef4f41a3b752cc50afddad`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:54.762875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1c8372f7-9284-4614-8ba5-e6b42ae17903 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:54.789023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:54.848515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1099/2196] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:54 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:54.855260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51aa772-d07d-4a6e-a711-219cbc507879 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:54.928130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51aa772-d07d-4a6e-a711-219cbc507879 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:54.934213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b230939-de49-47ac-bc59-3e32d7941bb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:54.935014 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b230939-de49-47ac-bc59-3e32d7941bb8 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:54.935306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b230939-de49-47ac-bc59-3e32d7941bb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:54.935633 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b230939-de49-47ac-bc59-3e32d7941bb8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:54.936381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b230939-de49-47ac-bc59-3e32d7941bb8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:54.939225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b230939-de49-47ac-bc59-3e32d7941bb8 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.039902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1098/2197] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:54 2022] POST /identity/v3/projects => generated 365 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:55.043767 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8372f7-9284-4614-8ba5-e6b42ae17903 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.054954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8372f7-9284-4614-8ba5-e6b42ae17903 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:55.118970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff1e8d-a2ab-49f9-9482-2967665516d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:55.119648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff1e8d-a2ab-49f9-9482-2967665516d1 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.119931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff1e8d-a2ab-49f9-9482-2967665516d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.120210 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfff1e8d-a2ab-49f9-9482-2967665516d1 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.120966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfff1e8d-a2ab-49f9-9482-2967665516d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:55.122977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfff1e8d-a2ab-49f9-9482-2967665516d1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.221295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1100/2198] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:55 2022] POST /identity/v3/users => generated 289 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:55.222677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b230939-de49-47ac-bc59-3e32d7941bb8 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.224657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0b230939-de49-47ac-bc59-3e32d7941bb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:55.225248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff1e8d-a2ab-49f9-9482-2967665516d1 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.235719 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfff1e8d-a2ab-49f9-9482-2967665516d1 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:55.238383 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b230939-de49-47ac-bc59-3e32d7941bb8 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:55.240504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f1e4e6-5dca-475f-81c2-6f3dd3c70422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:55.241217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f1e4e6-5dca-475f-81c2-6f3dd3c70422 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.241855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f1e4e6-5dca-475f-81c2-6f3dd3c70422 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.242140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f1e4e6-5dca-475f-81c2-6f3dd3c70422 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.243362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f1e4e6-5dca-475f-81c2-6f3dd3c70422 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:55.244230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:55.244496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f1e4e6-5dca-475f-81c2-6f3dd3c70422 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.245332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.245707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.246721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] PATH_INFO: `/v3/projects/2607c07d53c74bcf9abf439950317789` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.248036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2607c07d53c74bcf9abf439950317789)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:55.249973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.261347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1101/2199] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:55 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:55.262477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f1e4e6-5dca-475f-81c2-6f3dd3c70422 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.277796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f1e4e6-5dca-475f-81c2-6f3dd3c70422 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:55.284451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:DELETE=2,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:55.288896 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecab0fc-608f-4008-9346-996905618239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:55.290361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecab0fc-608f-4008-9346-996905618239 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.290838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecab0fc-608f-4008-9346-996905618239 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.291182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eecab0fc-608f-4008-9346-996905618239 admin admin] PATH_INFO: `/v3/projects/e64f00671ed949d59e5175632f106046/users/52d7fa21973c44b4b95a804091c74604/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.315105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecab0fc-608f-4008-9346-996905618239 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e64f00671ed949d59e5175632f106046, user_id=52d7fa21973c44b4b95a804091c74604, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:55.325017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eecab0fc-608f-4008-9346-996905618239 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.418878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1102/2200] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:55 2022] PUT /identity/v3/projects/e64f00671ed949d59e5175632f106046/users/52d7fa21973c44b4b95a804091c74604/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:06:55.422778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecab0fc-608f-4008-9346-996905618239 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.424609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2607c07d53c74bcf9abf439950317789', 'request_id': 'req-5aa77834-58f2-45f0-9d39-894f5b385f14'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:55.498850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eecab0fc-608f-4008-9346-996905618239 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:55.506345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18506f26-6184-4413-ad6f-17af664d45f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:55.507231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18506f26-6184-4413-ad6f-17af664d45f9 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.507810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18506f26-6184-4413-ad6f-17af664d45f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.508152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18506f26-6184-4413-ad6f-17af664d45f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.508926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18506f26-6184-4413-ad6f-17af664d45f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:55.510580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18506f26-6184-4413-ad6f-17af664d45f9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.517396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2607c07d53c74bcf9abf439950317789', 'request_id': 'req-5aa77834-58f2-45f0-9d39-894f5b385f14'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:55.524977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1103/2201] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:55 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:06:55.525845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18506f26-6184-4413-ad6f-17af664d45f9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.526570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2607c07d53c74bcf9abf439950317789', 'request_id': 'req-5aa77834-58f2-45f0-9d39-894f5b385f14'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:55.537524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18506f26-6184-4413-ad6f-17af664d45f9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:55.537703 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] The token cache is being invalidate because project 2607c07d53c74bcf9abf439950317789 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:06:55.537942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:55.538675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5aa77834-58f2-45f0-9d39-894f5b385f14 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:06:55.540660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1099/2202] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:06:55 2022] DELETE /identity/v3/projects/2607c07d53c74bcf9abf439950317789 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:55.546645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244ee37f-7e86-4726-8745-96d5dfb8e84e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.549313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244ee37f-7e86-4726-8745-96d5dfb8e84e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.549313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244ee37f-7e86-4726-8745-96d5dfb8e84e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.549523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7713772-1eaa-45e4-bfc1-ab31933f27c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:55.549523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7713772-1eaa-45e4-bfc1-ab31933f27c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.549523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7713772-1eaa-45e4-bfc1-ab31933f27c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.549523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7713772-1eaa-45e4-bfc1-ab31933f27c1 admin admin] PATH_INFO: `/v3/projects/e64f00671ed949d59e5175632f106046/users/52d7fa21973c44b4b95a804091c74604/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.551184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7713772-1eaa-45e4-bfc1-ab31933f27c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e64f00671ed949d59e5175632f106046, user_id=52d7fa21973c44b4b95a804091c74604, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:55.554133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7713772-1eaa-45e4-bfc1-ab31933f27c1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.565840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1104/2203] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:55 2022] PUT /identity/v3/projects/e64f00671ed949d59e5175632f106046/users/52d7fa21973c44b4b95a804091c74604/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:55.566656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-244ee37f-7e86-4726-8745-96d5dfb8e84e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:55.567957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-244ee37f-7e86-4726-8745-96d5dfb8e84e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:55.579658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6e9c55-8b33-417f-8b59-159a664da27f None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.580219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6e9c55-8b33-417f-8b59-159a664da27f None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.580938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6e9c55-8b33-417f-8b59-159a664da27f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.618647 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4a6e9c55-8b33-417f-8b59-159a664da27f None None] MFA Rules not processed for user `52d7fa21973c44b4b95a804091c74604`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:55.619883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4a6e9c55-8b33-417f-8b59-159a664da27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:55.671444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1100/2204] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:55.678893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e998b7-a83e-42da-a8d4-c742af1fb508 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.679436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e998b7-a83e-42da-a8d4-c742af1fb508 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.680012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37e998b7-a83e-42da-a8d4-c742af1fb508 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.710171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-37e998b7-a83e-42da-a8d4-c742af1fb508 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:55.712568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-37e998b7-a83e-42da-a8d4-c742af1fb508 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:55.729053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1101/2205] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:55.734363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e998b7-a83e-42da-a8d4-c742af1fb508 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.737199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1105/2206] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:55 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:55.753051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6e9c55-8b33-417f-8b59-159a664da27f None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.754573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4a6e9c55-8b33-417f-8b59-159a664da27f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:55.755807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37e998b7-a83e-42da-a8d4-c742af1fb508 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:55.762627 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fb581b-da06-4214-a72b-abc335cefce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:55.763334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fb581b-da06-4214-a72b-abc335cefce9 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.763804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fb581b-da06-4214-a72b-abc335cefce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.764087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fb581b-da06-4214-a72b-abc335cefce9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.764893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fb581b-da06-4214-a72b-abc335cefce9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:06:55.765227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fb581b-da06-4214-a72b-abc335cefce9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:55.767132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2fb581b-da06-4214-a72b-abc335cefce9 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.783464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1102/2207] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:06:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:06:55.788908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fb581b-da06-4214-a72b-abc335cefce9 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.808754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2fb581b-da06-4214-a72b-abc335cefce9 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:55.815154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10abd86-2822-4d4c-8a71-d90bfd2de9e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:55.816009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10abd86-2822-4d4c-8a71-d90bfd2de9e1 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.816422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10abd86-2822-4d4c-8a71-d90bfd2de9e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.816713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10abd86-2822-4d4c-8a71-d90bfd2de9e1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.817502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10abd86-2822-4d4c-8a71-d90bfd2de9e1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:55.819850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10abd86-2822-4d4c-8a71-d90bfd2de9e1 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.836642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6e9c55-8b33-417f-8b59-159a664da27f None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:55.843887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2361cd53-79dd-4536-aa05-181bde5ba9be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:55.844549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361cd53-79dd-4536-aa05-181bde5ba9be service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:55.844862 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361cd53-79dd-4536-aa05-181bde5ba9be service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:55.845318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2361cd53-79dd-4536-aa05-181bde5ba9be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:55.849730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:55.864556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2361cd53-79dd-4536-aa05-181bde5ba9be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:55.866102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2361cd53-79dd-4536-aa05-181bde5ba9be service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:55.882436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1106/2208] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:55 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:55.908336 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1103/2209] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:55 2022] POST /identity/v3/projects => generated 373 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:55.912452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2361cd53-79dd-4536-aa05-181bde5ba9be service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:55.926436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2361cd53-79dd-4536-aa05-181bde5ba9be service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:56.004404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7c1cb4-8ecb-414a-bfc2-79a0e0378871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:56.005158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7c1cb4-8ecb-414a-bfc2-79a0e0378871 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:56.005838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7c1cb4-8ecb-414a-bfc2-79a0e0378871 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:56.005838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7c1cb4-8ecb-414a-bfc2-79a0e0378871 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:56.006452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7c1cb4-8ecb-414a-bfc2-79a0e0378871 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:56.008390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7c1cb4-8ecb-414a-bfc2-79a0e0378871 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:56.099721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1107/2210] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:55 2022] POST /identity/v3/users => generated 287 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:06:56.103394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10abd86-2822-4d4c-8a71-d90bfd2de9e1 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:56.109160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7c1cb4-8ecb-414a-bfc2-79a0e0378871 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:56.112291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9a7c1cb4-8ecb-414a-bfc2-79a0e0378871 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:56.115647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10abd86-2822-4d4c-8a71-d90bfd2de9e1 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:56.121904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8c8e80-14ce-4215-90d6-9f0d9315eec3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:56.123379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8c8e80-14ce-4215-90d6-9f0d9315eec3 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:56.123808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8c8e80-14ce-4215-90d6-9f0d9315eec3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:56.124385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8c8e80-14ce-4215-90d6-9f0d9315eec3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:56.125461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8c8e80-14ce-4215-90d6-9f0d9315eec3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:56.129811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b8c8e80-14ce-4215-90d6-9f0d9315eec3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:56.143829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1104/2211] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:56 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:56.147919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8c8e80-14ce-4215-90d6-9f0d9315eec3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:56.159707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8c8e80-14ce-4215-90d6-9f0d9315eec3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:56.165543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63d2ceb-33a5-43e7-9db9-62d9aa3f37a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:56.166274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63d2ceb-33a5-43e7-9db9-62d9aa3f37a3 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:56.166623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63d2ceb-33a5-43e7-9db9-62d9aa3f37a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:56.167127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63d2ceb-33a5-43e7-9db9-62d9aa3f37a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:56.167911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63d2ceb-33a5-43e7-9db9-62d9aa3f37a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:56.170859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d63d2ceb-33a5-43e7-9db9-62d9aa3f37a3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:56.176077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7c1cb4-8ecb-414a-bfc2-79a0e0378871 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:56.180163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac5fa8f-b57c-4cef-86cc-ed033c267357 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:56.180833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5fa8f-b57c-4cef-86cc-ed033c267357 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:56.181120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5fa8f-b57c-4cef-86cc-ed033c267357 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:56.181399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac5fa8f-b57c-4cef-86cc-ed033c267357 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:56.182261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1105/2212] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:56 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:06:56.183403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-bac5fa8f-b57c-4cef-86cc-ed033c267357 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:56.186102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63d2ceb-33a5-43e7-9db9-62d9aa3f37a3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:56.202566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63d2ceb-33a5-43e7-9db9-62d9aa3f37a3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:56.208236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c76291-1ff6-4cc6-aaf0-cd7811a85d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:56.209395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c76291-1ff6-4cc6-aaf0-cd7811a85d25 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:56.209777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c76291-1ff6-4cc6-aaf0-cd7811a85d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:56.210155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c76291-1ff6-4cc6-aaf0-cd7811a85d25 admin admin] PATH_INFO: `/v3/projects/dc66981117554bff8ccd32bcdb1161b6/users/9a8382f4d8ad4acaa7402af0ca82027e/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:56.233141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c76291-1ff6-4cc6-aaf0-cd7811a85d25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc66981117554bff8ccd32bcdb1161b6, user_id=9a8382f4d8ad4acaa7402af0ca82027e, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:56.238811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c76291-1ff6-4cc6-aaf0-cd7811a85d25 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:56.262391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac5fa8f-b57c-4cef-86cc-ed033c267357 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:56.264233 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac5fa8f-b57c-4cef-86cc-ed033c267357 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:56.283630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1108/2213] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:56 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:56.344053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1106/2214] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:56 2022] PUT /identity/v3/projects/dc66981117554bff8ccd32bcdb1161b6/users/9a8382f4d8ad4acaa7402af0ca82027e/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:06:56.354649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f706b-0f54-44d0-98b2-870759391782 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:56.354845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f706b-0f54-44d0-98b2-870759391782 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:56.355126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584f706b-0f54-44d0-98b2-870759391782 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:56.394799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-584f706b-0f54-44d0-98b2-870759391782 None None] MFA Rules not processed for user `9a8382f4d8ad4acaa7402af0ca82027e`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:56.396389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-584f706b-0f54-44d0-98b2-870759391782 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:56.507963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1109/2215] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:56 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:56.513380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c76291-1ff6-4cc6-aaf0-cd7811a85d25 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:56.605436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c76291-1ff6-4cc6-aaf0-cd7811a85d25 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:56.611716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c237ba28-60bb-4bad-9bda-75cc948c8141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:56.612811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237ba28-60bb-4bad-9bda-75cc948c8141 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:56.613210 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237ba28-60bb-4bad-9bda-75cc948c8141 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:56.613785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c237ba28-60bb-4bad-9bda-75cc948c8141 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:56.614717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c237ba28-60bb-4bad-9bda-75cc948c8141 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:56.617280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c237ba28-60bb-4bad-9bda-75cc948c8141 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:56.766363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1107/2216] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:06:56 2022] POST /identity/v3/projects => generated 375 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:56.771503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584f706b-0f54-44d0-98b2-870759391782 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:56.799567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584f706b-0f54-44d0-98b2-870759391782 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:56.807992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daf20cd-6985-4735-bb83-94266ae523c9 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:56.808482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daf20cd-6985-4735-bb83-94266ae523c9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:56.809013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daf20cd-6985-4735-bb83-94266ae523c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:56.847595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9daf20cd-6985-4735-bb83-94266ae523c9 None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:56.849093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9daf20cd-6985-4735-bb83-94266ae523c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:56.932222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1108/2217] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:06:56 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:56.969546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b4c9d3-a7fb-476e-9d07-225fa16ec1ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:56.970278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b4c9d3-a7fb-476e-9d07-225fa16ec1ce admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:56.970552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b4c9d3-a7fb-476e-9d07-225fa16ec1ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:56.971119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b4c9d3-a7fb-476e-9d07-225fa16ec1ce admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:56.971523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b4c9d3-a7fb-476e-9d07-225fa16ec1ce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:56.973701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b4c9d3-a7fb-476e-9d07-225fa16ec1ce admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:56.999183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daf20cd-6985-4735-bb83-94266ae523c9 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:57.000873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9daf20cd-6985-4735-bb83-94266ae523c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:57.093290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daf20cd-6985-4735-bb83-94266ae523c9 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:57.101734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1110/2218] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:06:56 2022] POST /identity/v3/users => generated 294 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:06:57.102778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f852d5b-2646-4e5b-91dc-db8b0fb0613b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:57.103492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f852d5b-2646-4e5b-91dc-db8b0fb0613b service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:57.103801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f852d5b-2646-4e5b-91dc-db8b0fb0613b service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:57.104081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f852d5b-2646-4e5b-91dc-db8b0fb0613b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:57.110845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b4c9d3-a7fb-476e-9d07-225fa16ec1ce admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:57.113571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f852d5b-2646-4e5b-91dc-db8b0fb0613b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:57.115191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f852d5b-2646-4e5b-91dc-db8b0fb0613b service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:57.123520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b4c9d3-a7fb-476e-9d07-225fa16ec1ce admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:57.126379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1109/2219] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:06:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jun 10 13:06:57.131152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:57.134538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf816de1-95fd-402d-9cf3-de8e468dc320 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:57.135280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf816de1-95fd-402d-9cf3-de8e468dc320 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:57.135577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf816de1-95fd-402d-9cf3-de8e468dc320 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:57.135858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf816de1-95fd-402d-9cf3-de8e468dc320 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:57.136751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf816de1-95fd-402d-9cf3-de8e468dc320 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:57.139847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf816de1-95fd-402d-9cf3-de8e468dc320 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:57.195068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1111/2220] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:57 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:57.195997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f852d5b-2646-4e5b-91dc-db8b0fb0613b service swift] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:57.222873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f852d5b-2646-4e5b-91dc-db8b0fb0613b service swift] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:57.236089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745920f-7aee-42c0-8675-9934429d9e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:57.236089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745920f-7aee-42c0-8675-9934429d9e65 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:57.236089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745920f-7aee-42c0-8675-9934429d9e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:57.236089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745920f-7aee-42c0-8675-9934429d9e65 admin admin] PATH_INFO: `/v3/projects/34f3af60dbfd46afa6f9996d0c08580b/users/98db95263010497fad65277e5d773aaa/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:57.267602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3745920f-7aee-42c0-8675-9934429d9e65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=34f3af60dbfd46afa6f9996d0c08580b, user_id=98db95263010497fad65277e5d773aaa, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:57.271053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3745920f-7aee-42c0-8675-9934429d9e65 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:57.283478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:57.361967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1110/2221] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:57 2022] PUT /identity/v3/projects/34f3af60dbfd46afa6f9996d0c08580b/users/98db95263010497fad65277e5d773aaa/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:57.366038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf816de1-95fd-402d-9cf3-de8e468dc320 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:57.378844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf816de1-95fd-402d-9cf3-de8e468dc320 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:57.521568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4630e3a-3a6d-4ead-8e61-4305696eb72a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:57.522372 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4630e3a-3a6d-4ead-8e61-4305696eb72a admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:57.524317 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4630e3a-3a6d-4ead-8e61-4305696eb72a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:57.524619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4630e3a-3a6d-4ead-8e61-4305696eb72a admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:57.525532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4630e3a-3a6d-4ead-8e61-4305696eb72a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:57.527223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4630e3a-3a6d-4ead-8e61-4305696eb72a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:57.566335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1112/2222] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:06:57 2022] GET /identity/v3/roles => generated 1420 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:06:57.573553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745920f-7aee-42c0-8675-9934429d9e65 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:57.609303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745920f-7aee-42c0-8675-9934429d9e65 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:57.622805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ef46db-bd52-4f93-9536-63eab56fec76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:57.625182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ef46db-bd52-4f93-9536-63eab56fec76 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:57.625182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ef46db-bd52-4f93-9536-63eab56fec76 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:57.625182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ef46db-bd52-4f93-9536-63eab56fec76 admin admin] PATH_INFO: `/v3/projects/34f3af60dbfd46afa6f9996d0c08580b/users/98db95263010497fad65277e5d773aaa/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:57.630773 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ef46db-bd52-4f93-9536-63eab56fec76 admin admin] RBAC: Authorizing `identity:create_grant(project_id=34f3af60dbfd46afa6f9996d0c08580b, user_id=98db95263010497fad65277e5d773aaa, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:57.635536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ef46db-bd52-4f93-9536-63eab56fec76 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:57.653946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1111/2223] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:06:57 2022] PUT /identity/v3/projects/34f3af60dbfd46afa6f9996d0c08580b/users/98db95263010497fad65277e5d773aaa/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:06:57.663292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57bfdf-7f09-4b3a-9205-ac2f2c7d9bc9 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:57.663580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57bfdf-7f09-4b3a-9205-ac2f2c7d9bc9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:57.663860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a57bfdf-7f09-4b3a-9205-ac2f2c7d9bc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:57.693053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4a57bfdf-7f09-4b3a-9205-ac2f2c7d9bc9 None None] MFA Rules not processed for user `98db95263010497fad65277e5d773aaa`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:57.694250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4a57bfdf-7f09-4b3a-9205-ac2f2c7d9bc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:57.785968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1113/2224] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:06:57 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:57.834871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ef46db-bd52-4f93-9536-63eab56fec76 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:57.897543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ef46db-bd52-4f93-9536-63eab56fec76 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:57.904334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc05c499-af50-438f-ac9d-ce0d672014b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:57.905160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc05c499-af50-438f-ac9d-ce0d672014b0 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:57.905786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc05c499-af50-438f-ac9d-ce0d672014b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:57.906200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc05c499-af50-438f-ac9d-ce0d672014b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:57.990441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc05c499-af50-438f-ac9d-ce0d672014b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:57.995474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc05c499-af50-438f-ac9d-ce0d672014b0 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:58.018134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1112/2225] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:06:57 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 186 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:06:58.146028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:58.329476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:59.029781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6d8fbb-f53b-470b-bd57-25ac863062ed None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:59.030072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6d8fbb-f53b-470b-bd57-25ac863062ed None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:59.030353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f6d8fbb-f53b-470b-bd57-25ac863062ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:59.048139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-8f6d8fbb-f53b-470b-bd57-25ac863062ed None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:59.049418 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8f6d8fbb-f53b-470b-bd57-25ac863062ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:59.104946 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1114/2226] 10.222.0.25 () {58 vars in 967 bytes} [Fri Jun 10 13:06:59 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:06:59.119356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc05c499-af50-438f-ac9d-ce0d672014b0 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:59.192329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc05c499-af50-438f-ac9d-ce0d672014b0 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:59.198530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925516a2-a263-45c3-b22b-ac50be414e6f service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:59.199416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925516a2-a263-45c3-b22b-ac50be414e6f service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:59.199702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925516a2-a263-45c3-b22b-ac50be414e6f service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:59.200012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925516a2-a263-45c3-b22b-ac50be414e6f service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:59.209135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925516a2-a263-45c3-b22b-ac50be414e6f service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:59.214105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925516a2-a263-45c3-b22b-ac50be414e6f service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:59.220606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1113/2227] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:06:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 13:06:59.620829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:06:59.874169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85685c7f-0666-492d-bb9d-04f184cdb76d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:59.877693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85685c7f-0666-492d-bb9d-04f184cdb76d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:59.877693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85685c7f-0666-492d-bb9d-04f184cdb76d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:59.897798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-85685c7f-0666-492d-bb9d-04f184cdb76d None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:06:59.899747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-85685c7f-0666-492d-bb9d-04f184cdb76d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:06:59.925147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925516a2-a263-45c3-b22b-ac50be414e6f service swift] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:06:59.932359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1115/2228] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:06:59 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:06:59.938861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925516a2-a263-45c3-b22b-ac50be414e6f service swift] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:06:59.946769 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca3fdc3-ed8c-48df-9a1d-b4c1c0af9685 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:06:59.947839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca3fdc3-ed8c-48df-9a1d-b4c1c0af9685 service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:06:59.948179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca3fdc3-ed8c-48df-9a1d-b4c1c0af9685 service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:06:59.948565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca3fdc3-ed8c-48df-9a1d-b4c1c0af9685 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:06:59.958824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca3fdc3-ed8c-48df-9a1d-b4c1c0af9685 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:06:59.960681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bca3fdc3-ed8c-48df-9a1d-b4c1c0af9685 service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:06:59.977745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1114/2229] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:06:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jun 10 13:07:00.254756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:00.402793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:00.663042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83ef26-5fe4-41e4-ad23-02ae21c3d129 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:00.663538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83ef26-5fe4-41e4-ad23-02ae21c3d129 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:00.663627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f83ef26-5fe4-41e4-ad23-02ae21c3d129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:00.689302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4f83ef26-5fe4-41e4-ad23-02ae21c3d129 None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:00.710985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4f83ef26-5fe4-41e4-ad23-02ae21c3d129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:00.710985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1116/2230] 10.222.0.25 () {58 vars in 967 bytes} [Fri Jun 10 13:07:00 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:00.719682 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca3fdc3-ed8c-48df-9a1d-b4c1c0af9685 service swift] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:00.733317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca3fdc3-ed8c-48df-9a1d-b4c1c0af9685 service swift] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:00.750167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea689a9-159f-4fe8-b7cb-be0e8f152018 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:00.750921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea689a9-159f-4fe8-b7cb-be0e8f152018 service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:00.751212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea689a9-159f-4fe8-b7cb-be0e8f152018 service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:00.754675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea689a9-159f-4fe8-b7cb-be0e8f152018 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:00.760413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea689a9-159f-4fe8-b7cb-be0e8f152018 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:00.762152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea689a9-159f-4fe8-b7cb-be0e8f152018 service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:00.770278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1115/2231] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:07:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 13:07:01.441451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f83ef26-5fe4-41e4-ad23-02ae21c3d129 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:01.539132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f83ef26-5fe4-41e4-ad23-02ae21c3d129 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:01.543729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:01.544680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:01.544983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:01.545255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] PATH_INFO: `/v3/users/9a8382f4d8ad4acaa7402af0ca82027e` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:01.546459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9a8382f4d8ad4acaa7402af0ca82027e)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:01.548375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:01.708629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9a8382f4d8ad4acaa7402af0ca82027e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:01.709323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9a8382f4d8ad4acaa7402af0ca82027e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:01.722577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9a8382f4d8ad4acaa7402af0ca82027e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:01.781486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9a8382f4d8ad4acaa7402af0ca82027e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:01.782433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9a8382f4d8ad4acaa7402af0ca82027e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:01.802199 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1117/2232] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:01 2022] DELETE /identity/v3/users/9a8382f4d8ad4acaa7402af0ca82027e => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:01.806735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04f401a-66a6-4fa0-9b5f-cd38a6e97e52 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:01.807519 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04f401a-66a6-4fa0-9b5f-cd38a6e97e52 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:01.807519 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04f401a-66a6-4fa0-9b5f-cd38a6e97e52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:01.824837 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-f04f401a-66a6-4fa0-9b5f-cd38a6e97e52 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:01.826583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f04f401a-66a6-4fa0-9b5f-cd38a6e97e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:01.901890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1116/2233] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:01.908523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:01.910038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:01.989865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6069837-d02a-40ca-b321-6455e637a368 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:01.994498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e5dc2b-07da-48b2-aa2c-30ab6dda37ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:01.994662 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e5dc2b-07da-48b2-aa2c-30ab6dda37ab service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:01.994950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e5dc2b-07da-48b2-aa2c-30ab6dda37ab service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:01.995234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e5dc2b-07da-48b2-aa2c-30ab6dda37ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:02.007159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e5dc2b-07da-48b2-aa2c-30ab6dda37ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:02.008723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e5dc2b-07da-48b2-aa2c-30ab6dda37ab service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:02.020351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1118/2234] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:02.415227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:DELETE=16,keystone:SELECT=82,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:02.460861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:02.737990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04f401a-66a6-4fa0-9b5f-cd38a6e97e52 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:02.739504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f04f401a-66a6-4fa0-9b5f-cd38a6e97e52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:02.753552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04f401a-66a6-4fa0-9b5f-cd38a6e97e52 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:02.757316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:02.758268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:02.758550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:02.758823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] PATH_INFO: `/v3/projects/dc66981117554bff8ccd32bcdb1161b6` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:02.760004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc66981117554bff8ccd32bcdb1161b6)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:02.762047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:02.878800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc66981117554bff8ccd32bcdb1161b6', 'request_id': 'req-659f54fd-4cda-42d9-9a05-583c8222a3e0'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:02.991717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc66981117554bff8ccd32bcdb1161b6', 'request_id': 'req-659f54fd-4cda-42d9-9a05-583c8222a3e0'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:02.998717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc66981117554bff8ccd32bcdb1161b6', 'request_id': 'req-659f54fd-4cda-42d9-9a05-583c8222a3e0'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:03.009390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] The token cache is being invalidate because project dc66981117554bff8ccd32bcdb1161b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:03.009686 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:03.010349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:03.011910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1117/2235] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:02 2022] DELETE /identity/v3/projects/dc66981117554bff8ccd32bcdb1161b6 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:03.016176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e5dc2b-07da-48b2-aa2c-30ab6dda37ab service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:03.025729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e5dc2b-07da-48b2-aa2c-30ab6dda37ab service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:03.082712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:03.083587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:03.083873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:03.084154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] PATH_INFO: `/v3/users/98db95263010497fad65277e5d773aaa` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:03.085400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=98db95263010497fad65277e5d773aaa)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:03.087593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:03.303223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98db95263010497fad65277e5d773aaa'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:03.303929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98db95263010497fad65277e5d773aaa'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:03.311138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98db95263010497fad65277e5d773aaa'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:03.362750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98db95263010497fad65277e5d773aaa'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:03.363493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6101274c-3f6f-4b0a-9234-09c4b77c6cd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98db95263010497fad65277e5d773aaa'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:03.381356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1119/2236] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:03 2022] DELETE /identity/v3/users/98db95263010497fad65277e5d773aaa => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:03.907419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:03.909804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:03.927378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f54fd-4cda-42d9-9a05-583c8222a3e0 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:04.000023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:04.006499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.006499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.006499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] PATH_INFO: `/v3/projects/34f3af60dbfd46afa6f9996d0c08580b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:04.006499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=34f3af60dbfd46afa6f9996d0c08580b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:04.006499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:04.185860 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '34f3af60dbfd46afa6f9996d0c08580b', 'request_id': 'req-e4686862-3e56-4d0c-9c7c-568d9c110f7c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:04.265989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '34f3af60dbfd46afa6f9996d0c08580b', 'request_id': 'req-e4686862-3e56-4d0c-9c7c-568d9c110f7c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:04.268626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:04.272797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '34f3af60dbfd46afa6f9996d0c08580b', 'request_id': 'req-e4686862-3e56-4d0c-9c7c-568d9c110f7c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:04.283935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] The token cache is being invalidate because project 34f3af60dbfd46afa6f9996d0c08580b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:04.284229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:04.285123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e4686862-3e56-4d0c-9c7c-568d9c110f7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:04.286836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1118/2237] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:03 2022] DELETE /identity/v3/projects/34f3af60dbfd46afa6f9996d0c08580b => generated 0 bytes in 380 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:04.291869 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeed885-9faa-4eef-bd86-d8cfc90d53f8 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.292549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeed885-9faa-4eef-bd86-d8cfc90d53f8 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.292861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adeed885-9faa-4eef-bd86-d8cfc90d53f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:04.314352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-adeed885-9faa-4eef-bd86-d8cfc90d53f8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:04.315507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-adeed885-9faa-4eef-bd86-d8cfc90d53f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:04.351783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:04.392554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1120/2238] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:04.400340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9be022-a786-4209-a00a-e76774728f26 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.405239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9be022-a786-4209-a00a-e76774728f26 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.405239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9be022-a786-4209-a00a-e76774728f26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:04.435643 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9c9be022-a786-4209-a00a-e76774728f26 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:04.436953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9c9be022-a786-4209-a00a-e76774728f26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:04.446115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1119/2239] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:04.454933 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeed885-9faa-4eef-bd86-d8cfc90d53f8 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:04.465037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adeed885-9faa-4eef-bd86-d8cfc90d53f8 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:04.470827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7b7b6d-3204-4221-92d9-b81ab3782ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:04.471091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7b7b6d-3204-4221-92d9-b81ab3782ddb admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.471320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7b7b6d-3204-4221-92d9-b81ab3782ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.471600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7b7b6d-3204-4221-92d9-b81ab3782ddb admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:04.472344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7b7b6d-3204-4221-92d9-b81ab3782ddb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:07:04.472630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7b7b6d-3204-4221-92d9-b81ab3782ddb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:04.474314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7b7b6d-3204-4221-92d9-b81ab3782ddb admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:04.488593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1121/2240] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:07:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:07:04.492953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9be022-a786-4209-a00a-e76774728f26 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:04.502324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9be022-a786-4209-a00a-e76774728f26 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:04.507308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d441fd-3a12-4aa4-b639-3e2b8fd67f9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:04.507960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d441fd-3a12-4aa4-b639-3e2b8fd67f9e admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.508243 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d441fd-3a12-4aa4-b639-3e2b8fd67f9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.508518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d441fd-3a12-4aa4-b639-3e2b8fd67f9e admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:04.509325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d441fd-3a12-4aa4-b639-3e2b8fd67f9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:04.511231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d441fd-3a12-4aa4-b639-3e2b8fd67f9e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:04.587928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1120/2241] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:07:04 2022] POST /identity/v3/projects => generated 373 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:07:04.592693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7b7b6d-3204-4221-92d9-b81ab3782ddb admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:04.602733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7b7b6d-3204-4221-92d9-b81ab3782ddb admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:04.658359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b081783-9c0a-4b26-ab53-3f41eef799e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:04.659378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b081783-9c0a-4b26-ab53-3f41eef799e1 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.660166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b081783-9c0a-4b26-ab53-3f41eef799e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.660681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b081783-9c0a-4b26-ab53-3f41eef799e1 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:04.661660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b081783-9c0a-4b26-ab53-3f41eef799e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:04.663615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b081783-9c0a-4b26-ab53-3f41eef799e1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:04.774878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1122/2242] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:07:04 2022] POST /identity/v3/users => generated 287 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:07:04.779784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d441fd-3a12-4aa4-b639-3e2b8fd67f9e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:04.790681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d441fd-3a12-4aa4-b639-3e2b8fd67f9e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:04.795542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbf1a46-b5c8-4c63-ae37-9b4e5ad3476a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:04.796206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbf1a46-b5c8-4c63-ae37-9b4e5ad3476a admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.796509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbf1a46-b5c8-4c63-ae37-9b4e5ad3476a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.796811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbf1a46-b5c8-4c63-ae37-9b4e5ad3476a admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:04.797690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbf1a46-b5c8-4c63-ae37-9b4e5ad3476a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:04.799467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbf1a46-b5c8-4c63-ae37-9b4e5ad3476a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:04.810753 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1121/2243] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:07:04.814700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b081783-9c0a-4b26-ab53-3f41eef799e1 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:04.825052 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b081783-9c0a-4b26-ab53-3f41eef799e1 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:04.830294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f880086c-9910-43f9-bd57-c5323eecf8de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:04.832678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880086c-9910-43f9-bd57-c5323eecf8de admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.832678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880086c-9910-43f9-bd57-c5323eecf8de admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.832678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f880086c-9910-43f9-bd57-c5323eecf8de admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:04.832678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f880086c-9910-43f9-bd57-c5323eecf8de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:04.834144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f880086c-9910-43f9-bd57-c5323eecf8de admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:04.846372 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1123/2244] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:04 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:07:04.850963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbf1a46-b5c8-4c63-ae37-9b4e5ad3476a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:04.862854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbf1a46-b5c8-4c63-ae37-9b4e5ad3476a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:04.871669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2a8e8-8a9a-440f-9d2f-f8fafda04670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:04.872800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2a8e8-8a9a-440f-9d2f-f8fafda04670 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.873006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2a8e8-8a9a-440f-9d2f-f8fafda04670 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.873377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87e2a8e8-8a9a-440f-9d2f-f8fafda04670 admin admin] PATH_INFO: `/v3/projects/0fdf6af645ba4a02a6a7278eb1fdff0d/users/24343309fb924981a37899f56f15b8ba/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:04.889485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e2a8e8-8a9a-440f-9d2f-f8fafda04670 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0fdf6af645ba4a02a6a7278eb1fdff0d, user_id=24343309fb924981a37899f56f15b8ba, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:04.892213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87e2a8e8-8a9a-440f-9d2f-f8fafda04670 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:04.985484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1122/2245] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:07:04 2022] PUT /identity/v3/projects/0fdf6af645ba4a02a6a7278eb1fdff0d/users/24343309fb924981a37899f56f15b8ba/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:04.987861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea3c5a-a606-4030-837c-ee850bf911cc None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:04.988321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea3c5a-a606-4030-837c-ee850bf911cc None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:04.988745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fea3c5a-a606-4030-837c-ee850bf911cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:05.009496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5fea3c5a-a606-4030-837c-ee850bf911cc None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:05.013886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5fea3c5a-a606-4030-837c-ee850bf911cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:05.082146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1124/2246] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:05.090265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2a8e8-8a9a-440f-9d2f-f8fafda04670 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:05.091822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-87e2a8e8-8a9a-440f-9d2f-f8fafda04670 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:05.155062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87e2a8e8-8a9a-440f-9d2f-f8fafda04670 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:05.159071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0cf254-14dd-455b-89bf-0cf66042a51c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:05.159726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0cf254-14dd-455b-89bf-0cf66042a51c service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:05.160018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0cf254-14dd-455b-89bf-0cf66042a51c service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:05.160356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0cf254-14dd-455b-89bf-0cf66042a51c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:05.170187 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0cf254-14dd-455b-89bf-0cf66042a51c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:05.171827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0cf254-14dd-455b-89bf-0cf66042a51c service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:05.181903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1123/2247] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:05.637432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:05.707157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=83,keystone:DELETE=12,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:06.096570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea3c5a-a606-4030-837c-ee850bf911cc None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:06.098063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5fea3c5a-a606-4030-837c-ee850bf911cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:06.114771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fea3c5a-a606-4030-837c-ee850bf911cc None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:06.120261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:06.121208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:06.121582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:06.121974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] PATH_INFO: `/v3/users/07363a81c7e844468ae3e2c203711d78` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:06.123174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] RBAC: Authorizing `identity:delete_user(user_id=07363a81c7e844468ae3e2c203711d78)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:06.125372 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:06.458799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '07363a81c7e844468ae3e2c203711d78'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.459486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '07363a81c7e844468ae3e2c203711d78'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.467541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '07363a81c7e844468ae3e2c203711d78'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.523616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0cf254-14dd-455b-89bf-0cf66042a51c service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:06.524189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0b0cf254-14dd-455b-89bf-0cf66042a51c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:06.534501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '07363a81c7e844468ae3e2c203711d78'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.535643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-cb167523-ec7c-4ca4-8765-a53d94b4281d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '07363a81c7e844468ae3e2c203711d78'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.537781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0cf254-14dd-455b-89bf-0cf66042a51c service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:06.543033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:06.543948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:06.544270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:06.544551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] PATH_INFO: `/v3/users/698dc89ee6ef4f41a3b752cc50afddad` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:06.545792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] RBAC: Authorizing `identity:delete_user(user_id=698dc89ee6ef4f41a3b752cc50afddad)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:06.547857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:06.599406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1125/2248] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:06 2022] DELETE /identity/v3/users/07363a81c7e844468ae3e2c203711d78 => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:06.605549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157c00ba-7136-442c-aeff-cbf7b33f35b3 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:06.606091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157c00ba-7136-442c-aeff-cbf7b33f35b3 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:06.606709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157c00ba-7136-442c-aeff-cbf7b33f35b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:06.635349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-157c00ba-7136-442c-aeff-cbf7b33f35b3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:06.636873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-157c00ba-7136-442c-aeff-cbf7b33f35b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:06.744205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1126/2249] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:06.746340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72ca4d1-e189-450f-851b-195506a3f0a4 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:06.746820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72ca4d1-e189-450f-851b-195506a3f0a4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:06.747281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72ca4d1-e189-450f-851b-195506a3f0a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:06.779033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '698dc89ee6ef4f41a3b752cc50afddad'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.780071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '698dc89ee6ef4f41a3b752cc50afddad'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.780910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '698dc89ee6ef4f41a3b752cc50afddad'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.790367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '698dc89ee6ef4f41a3b752cc50afddad'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.812218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e72ca4d1-e189-450f-851b-195506a3f0a4 None None] MFA Rules not processed for user `24343309fb924981a37899f56f15b8ba`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:06.813570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e72ca4d1-e189-450f-851b-195506a3f0a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:06.848394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '698dc89ee6ef4f41a3b752cc50afddad'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:06.882745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1124/2250] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:06 2022] DELETE /identity/v3/users/698dc89ee6ef4f41a3b752cc50afddad => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:07:06.883869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:06.887559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:06.917253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1127/2251] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:06 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:06.921026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f7871e-1d25-4839-9a33-782e02ff4772 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:06.924872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f7871e-1d25-4839-9a33-782e02ff4772 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:06.925802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f7871e-1d25-4839-9a33-782e02ff4772 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:06.957934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-b5f7871e-1d25-4839-9a33-782e02ff4772 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:06.959166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b5f7871e-1d25-4839-9a33-782e02ff4772 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:07.004993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6673430a-db58-4dc3-88f6-0dea3d792a62 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:07.012205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e130787d-eb8a-472d-ba2a-7de941128db7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:07.013864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e130787d-eb8a-472d-ba2a-7de941128db7 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:07.018718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e130787d-eb8a-472d-ba2a-7de941128db7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:07.018718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e130787d-eb8a-472d-ba2a-7de941128db7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:07.018718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e130787d-eb8a-472d-ba2a-7de941128db7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:07.095937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1128/2252] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:07.100316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f7871e-1d25-4839-9a33-782e02ff4772 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:07.101827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b5f7871e-1d25-4839-9a33-782e02ff4772 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:07.101998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e130787d-eb8a-472d-ba2a-7de941128db7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:07.103864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e130787d-eb8a-472d-ba2a-7de941128db7 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:07.121162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:07.123471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1125/2253] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 240 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:07.124337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e130787d-eb8a-472d-ba2a-7de941128db7 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:07.133312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e130787d-eb8a-472d-ba2a-7de941128db7 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:07.158276 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febe04a0-12bd-48fb-80c5-974d6734d45f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:07.158959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febe04a0-12bd-48fb-80c5-974d6734d45f service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:07.159393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febe04a0-12bd-48fb-80c5-974d6734d45f service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:07.159684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febe04a0-12bd-48fb-80c5-974d6734d45f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:07.171645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febe04a0-12bd-48fb-80c5-974d6734d45f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:07.173184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febe04a0-12bd-48fb-80c5-974d6734d45f service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:07.187791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1126/2254] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:07.211280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f7871e-1d25-4839-9a33-782e02ff4772 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:07.223007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26391f02-302c-4e1b-a7ea-2ccfc0b051b5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:07.223007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26391f02-302c-4e1b-a7ea-2ccfc0b051b5 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:07.223007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26391f02-302c-4e1b-a7ea-2ccfc0b051b5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:07.223007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26391f02-302c-4e1b-a7ea-2ccfc0b051b5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:07.291075 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26391f02-302c-4e1b-a7ea-2ccfc0b051b5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:07.292892 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26391f02-302c-4e1b-a7ea-2ccfc0b051b5 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:07.304355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1129/2255] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:07 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:07.498731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febe04a0-12bd-48fb-80c5-974d6734d45f service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:07.500168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-febe04a0-12bd-48fb-80c5-974d6734d45f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:07.523559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febe04a0-12bd-48fb-80c5-974d6734d45f service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:07.528530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:07.530729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:07.530890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:07.531659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] PATH_INFO: `/v3/projects/d1658a9d6de94030abaf2e7f5c82fc75` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:07.532998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d1658a9d6de94030abaf2e7f5c82fc75)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:07.535154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:07.621475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd1658a9d6de94030abaf2e7f5c82fc75', 'request_id': 'req-24b6b31a-825a-492d-96dc-e5f435015dc8'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:07.702227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26391f02-302c-4e1b-a7ea-2ccfc0b051b5 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:07.703971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-26391f02-302c-4e1b-a7ea-2ccfc0b051b5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:07.718949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26391f02-302c-4e1b-a7ea-2ccfc0b051b5 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:07.729263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd1658a9d6de94030abaf2e7f5c82fc75', 'request_id': 'req-24b6b31a-825a-492d-96dc-e5f435015dc8'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:07.742771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd1658a9d6de94030abaf2e7f5c82fc75', 'request_id': 'req-24b6b31a-825a-492d-96dc-e5f435015dc8'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:07.755047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] The token cache is being invalidate because project d1658a9d6de94030abaf2e7f5c82fc75 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:07.755913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:07.756860 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24b6b31a-825a-492d-96dc-e5f435015dc8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:07.758726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1127/2256] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:07 2022] DELETE /identity/v3/projects/d1658a9d6de94030abaf2e7f5c82fc75 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:07.763614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818c2c6-1275-4caf-8826-450cf4418d14 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:07.763907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818c2c6-1275-4caf-8826-450cf4418d14 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:07.764185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818c2c6-1275-4caf-8826-450cf4418d14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:07.782165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-6818c2c6-1275-4caf-8826-450cf4418d14 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:07.783372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6818c2c6-1275-4caf-8826-450cf4418d14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:07.795474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:07.796342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:07.796627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:07.796937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] PATH_INFO: `/v3/projects/5316f7d5300f4ca6823225ba1db23e96` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:07.798133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] RBAC: Authorizing `identity:delete_project(project_id=5316f7d5300f4ca6823225ba1db23e96)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:07.800163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:07.846227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1128/2257] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:07.853707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ab01b-62bc-47c8-b91c-28f1f3c5ad4c None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:07.854002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ab01b-62bc-47c8-b91c-28f1f3c5ad4c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:07.854350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211ab01b-62bc-47c8-b91c-28f1f3c5ad4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:07.873886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-211ab01b-62bc-47c8-b91c-28f1f3c5ad4c None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:07.875087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-211ab01b-62bc-47c8-b91c-28f1f3c5ad4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:07.885307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1129/2258] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:07.889841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211ab01b-62bc-47c8-b91c-28f1f3c5ad4c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:07.899049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-211ab01b-62bc-47c8-b91c-28f1f3c5ad4c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:07.903685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5fb419-809e-4ea4-a1dc-547f3cf9e685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:07.904383 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5fb419-809e-4ea4-a1dc-547f3cf9e685 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:07.904664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5fb419-809e-4ea4-a1dc-547f3cf9e685 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:07.904976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e5fb419-809e-4ea4-a1dc-547f3cf9e685 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:07.905700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5fb419-809e-4ea4-a1dc-547f3cf9e685 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:07:07.906031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5fb419-809e-4ea4-a1dc-547f3cf9e685 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:07.907636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e5fb419-809e-4ea4-a1dc-547f3cf9e685 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:07.921314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1130/2259] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:07:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:07:07.925526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5fb419-809e-4ea4-a1dc-547f3cf9e685 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:07.936919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e5fb419-809e-4ea4-a1dc-547f3cf9e685 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:07.942375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e59efbb-61af-453f-bfa1-49f93b0e1696 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:07.947468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e59efbb-61af-453f-bfa1-49f93b0e1696 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:07.947468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e59efbb-61af-453f-bfa1-49f93b0e1696 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:07.947468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e59efbb-61af-453f-bfa1-49f93b0e1696 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:07.947468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e59efbb-61af-453f-bfa1-49f93b0e1696 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:07.947468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e59efbb-61af-453f-bfa1-49f93b0e1696 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:07.963117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5316f7d5300f4ca6823225ba1db23e96', 'request_id': 'req-4feba2fb-ade3-40ce-9576-13cec9d4553f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:07.970487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5316f7d5300f4ca6823225ba1db23e96', 'request_id': 'req-4feba2fb-ade3-40ce-9576-13cec9d4553f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:07.979180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5316f7d5300f4ca6823225ba1db23e96', 'request_id': 'req-4feba2fb-ade3-40ce-9576-13cec9d4553f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:08.051226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] The token cache is being invalidate because project 5316f7d5300f4ca6823225ba1db23e96 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:08.051526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:08.052326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:08.052798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1131/2260] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:07:07 2022] POST /identity/v3/projects => generated 397 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:07:08.054213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1130/2261] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:07 2022] DELETE /identity/v3/projects/5316f7d5300f4ca6823225ba1db23e96 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:07:08.056816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e59efbb-61af-453f-bfa1-49f93b0e1696 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:08.057797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:08.061066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5e59efbb-61af-453f-bfa1-49f93b0e1696 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:08.061905 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:08.140674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:08.168264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4feba2fb-ade3-40ce-9576-13cec9d4553f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:08.174456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:08.175747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:08.176129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:08.176292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] PATH_INFO: `/v3/users/52d7fa21973c44b4b95a804091c74604` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:08.177660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] RBAC: Authorizing `identity:delete_user(user_id=52d7fa21973c44b4b95a804091c74604)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:08.177959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e59efbb-61af-453f-bfa1-49f93b0e1696 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:08.179656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:08.193524 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421c8b5e-41a1-4f78-8aca-60998a29a114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:08.194362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421c8b5e-41a1-4f78-8aca-60998a29a114 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:08.194761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421c8b5e-41a1-4f78-8aca-60998a29a114 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:08.195063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421c8b5e-41a1-4f78-8aca-60998a29a114 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:08.197533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421c8b5e-41a1-4f78-8aca-60998a29a114 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:08.199518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421c8b5e-41a1-4f78-8aca-60998a29a114 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:08.375482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1132/2262] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:07:08 2022] POST /identity/v3/users => generated 299 bytes in 319 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:07:08.379583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421c8b5e-41a1-4f78-8aca-60998a29a114 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:08.394431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421c8b5e-41a1-4f78-8aca-60998a29a114 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:08.399189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9148913-d5f6-4e6b-9dc4-8493ba46a2e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:08.399850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9148913-d5f6-4e6b-9dc4-8493ba46a2e5 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:08.400123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9148913-d5f6-4e6b-9dc4-8493ba46a2e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:08.400461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9148913-d5f6-4e6b-9dc4-8493ba46a2e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:08.401213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9148913-d5f6-4e6b-9dc4-8493ba46a2e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:08.402813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9148913-d5f6-4e6b-9dc4-8493ba46a2e5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:08.418523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1133/2263] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:08 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:07:08.421247 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9148913-d5f6-4e6b-9dc4-8493ba46a2e5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:08.431089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9148913-d5f6-4e6b-9dc4-8493ba46a2e5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:08.436222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a17c68-d140-4d48-82ef-f8a68da70bf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:08.436900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a17c68-d140-4d48-82ef-f8a68da70bf6 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:08.437185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a17c68-d140-4d48-82ef-f8a68da70bf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:08.437465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15a17c68-d140-4d48-82ef-f8a68da70bf6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:08.438218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a17c68-d140-4d48-82ef-f8a68da70bf6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:08.439820 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15a17c68-d140-4d48-82ef-f8a68da70bf6 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:08.450001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1134/2264] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:08 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:07:08.455688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a17c68-d140-4d48-82ef-f8a68da70bf6 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:08.465302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15a17c68-d140-4d48-82ef-f8a68da70bf6 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:08.469846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30672a30-9c5f-4793-9285-bb24b8df46e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:08.470694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30672a30-9c5f-4793-9285-bb24b8df46e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:08.470975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30672a30-9c5f-4793-9285-bb24b8df46e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:08.471248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30672a30-9c5f-4793-9285-bb24b8df46e1 admin admin] PATH_INFO: `/v3/projects/ef929d61f7494b5f81e0c15d2e47233c/users/99cb1cc352c144979ba4f12ff9c08522/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:08.486845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30672a30-9c5f-4793-9285-bb24b8df46e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ef929d61f7494b5f81e0c15d2e47233c, user_id=99cb1cc352c144979ba4f12ff9c08522, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:08.489719 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30672a30-9c5f-4793-9285-bb24b8df46e1 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:08.497087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52d7fa21973c44b4b95a804091c74604'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:08.498394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52d7fa21973c44b4b95a804091c74604'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:08.506137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52d7fa21973c44b4b95a804091c74604'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:08.602945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1135/2265] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:07:08 2022] PUT /identity/v3/projects/ef929d61f7494b5f81e0c15d2e47233c/users/99cb1cc352c144979ba4f12ff9c08522/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:07:08.614267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df688ec7-092a-4c61-8c81-a6a22be27b90 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:08.614557 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df688ec7-092a-4c61-8c81-a6a22be27b90 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:08.614869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df688ec7-092a-4c61-8c81-a6a22be27b90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:08.625768 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52d7fa21973c44b4b95a804091c74604'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:08.626415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52d7fa21973c44b4b95a804091c74604'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:08.641360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-df688ec7-092a-4c61-8c81-a6a22be27b90 None None] MFA Rules not processed for user `99cb1cc352c144979ba4f12ff9c08522`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:08.642582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-df688ec7-092a-4c61-8c81-a6a22be27b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:08.648261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1131/2266] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:08 2022] DELETE /identity/v3/users/52d7fa21973c44b4b95a804091c74604 => generated 0 bytes in 588 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:08.708813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1136/2267] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:08 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:08.720174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:08.722072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:08.781290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02c4954-bb7b-4867-a262-522973a8fdfe admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:08.784752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68ef3e4f-e663-442e-ae80-7607a6c64326 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:08.785435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ef3e4f-e663-442e-ae80-7607a6c64326 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:08.785979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ef3e4f-e663-442e-ae80-7607a6c64326 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:08.786260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68ef3e4f-e663-442e-ae80-7607a6c64326 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:08.795453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ef3e4f-e663-442e-ae80-7607a6c64326 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:08.797000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68ef3e4f-e663-442e-ae80-7607a6c64326 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:08.805677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1132/2268] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:08 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:09.151171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df688ec7-092a-4c61-8c81-a6a22be27b90 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:09.152955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-df688ec7-092a-4c61-8c81-a6a22be27b90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:09.253529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df688ec7-092a-4c61-8c81-a6a22be27b90 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:09.260897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:09.262849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:09.263834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:09.264127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] PATH_INFO: `/v3/projects/e64f00671ed949d59e5175632f106046` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:09.266320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] RBAC: Authorizing `identity:delete_project(project_id=e64f00671ed949d59e5175632f106046)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:09.269044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:09.463819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e64f00671ed949d59e5175632f106046', 'request_id': 'req-b5bdc48c-586c-4920-a076-302bfd8d2d4c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:09.592071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e64f00671ed949d59e5175632f106046', 'request_id': 'req-b5bdc48c-586c-4920-a076-302bfd8d2d4c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:09.598839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e64f00671ed949d59e5175632f106046', 'request_id': 'req-b5bdc48c-586c-4920-a076-302bfd8d2d4c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:09.611498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] The token cache is being invalidate because project e64f00671ed949d59e5175632f106046 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:09.619880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:09.619880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b5bdc48c-586c-4920-a076-302bfd8d2d4c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:09.619880 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1137/2269] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:09 2022] DELETE /identity/v3/projects/e64f00671ed949d59e5175632f106046 => generated 0 bytes in 464 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:07:09.624298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a407e1f3-2807-44b0-a33a-dcb4db383b6a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:09.624595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a407e1f3-2807-44b0-a33a-dcb4db383b6a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:09.624906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a407e1f3-2807-44b0-a33a-dcb4db383b6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:09.645510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a407e1f3-2807-44b0-a33a-dcb4db383b6a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:09.646720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a407e1f3-2807-44b0-a33a-dcb4db383b6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:09.709945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1133/2270] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:09.717625 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad38ed2e-1021-4e3f-9dfd-0fff61150d3a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:09.717939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad38ed2e-1021-4e3f-9dfd-0fff61150d3a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:09.723150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad38ed2e-1021-4e3f-9dfd-0fff61150d3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:09.750896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ad38ed2e-1021-4e3f-9dfd-0fff61150d3a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:09.750896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ad38ed2e-1021-4e3f-9dfd-0fff61150d3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:09.750896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1138/2271] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:09.754628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a407e1f3-2807-44b0-a33a-dcb4db383b6a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:09.765130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a407e1f3-2807-44b0-a33a-dcb4db383b6a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:09.769802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82630b3-cc4c-485e-b1c7-a245f4afd28b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:09.770439 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82630b3-cc4c-485e-b1c7-a245f4afd28b admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:09.770728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82630b3-cc4c-485e-b1c7-a245f4afd28b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:09.771006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82630b3-cc4c-485e-b1c7-a245f4afd28b admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:09.771735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82630b3-cc4c-485e-b1c7-a245f4afd28b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:07:09.772043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82630b3-cc4c-485e-b1c7-a245f4afd28b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:09.773790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82630b3-cc4c-485e-b1c7-a245f4afd28b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:09.788442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1134/2272] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:07:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:07:09.795513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad38ed2e-1021-4e3f-9dfd-0fff61150d3a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:09.804673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad38ed2e-1021-4e3f-9dfd-0fff61150d3a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:09.810427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b3a72f-ec97-4c07-8ca1-18537e7372c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:09.811126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b3a72f-ec97-4c07-8ca1-18537e7372c7 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:09.811416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b3a72f-ec97-4c07-8ca1-18537e7372c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:09.811722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b3a72f-ec97-4c07-8ca1-18537e7372c7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:09.812412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b3a72f-ec97-4c07-8ca1-18537e7372c7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:09.814487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01b3a72f-ec97-4c07-8ca1-18537e7372c7 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:09.821952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:09.966198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:09.976483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1139/2273] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:07:09 2022] POST /identity/v3/projects => generated 373 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:07:09.979611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82630b3-cc4c-485e-b1c7-a245f4afd28b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:09.990812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82630b3-cc4c-485e-b1c7-a245f4afd28b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:10.045844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ce48d-3383-407e-a7c3-04afa770efdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:10.046510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ce48d-3383-407e-a7c3-04afa770efdb admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:10.046793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ce48d-3383-407e-a7c3-04afa770efdb admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:10.047077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770ce48d-3383-407e-a7c3-04afa770efdb admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:10.047888 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770ce48d-3383-407e-a7c3-04afa770efdb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:10.049847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-770ce48d-3383-407e-a7c3-04afa770efdb admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:10.112759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1135/2274] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:07:09 2022] POST /identity/v3/users => generated 287 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:07:10.117027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b3a72f-ec97-4c07-8ca1-18537e7372c7 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:10.128027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b3a72f-ec97-4c07-8ca1-18537e7372c7 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:10.132688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c801f8-9704-4985-8120-d86636190175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:10.133372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c801f8-9704-4985-8120-d86636190175 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:10.133658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c801f8-9704-4985-8120-d86636190175 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:10.133939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c801f8-9704-4985-8120-d86636190175 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:10.134668 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c801f8-9704-4985-8120-d86636190175 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:10.136274 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c801f8-9704-4985-8120-d86636190175 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:10.149848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1140/2275] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:07:10.150733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ce48d-3383-407e-a7c3-04afa770efdb admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:10.160566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770ce48d-3383-407e-a7c3-04afa770efdb admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:10.165331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725448c4-06d1-443f-82ef-3e8aad554364 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:10.166045 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725448c4-06d1-443f-82ef-3e8aad554364 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:10.166333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725448c4-06d1-443f-82ef-3e8aad554364 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:10.166615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725448c4-06d1-443f-82ef-3e8aad554364 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:10.167398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725448c4-06d1-443f-82ef-3e8aad554364 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:10.169061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725448c4-06d1-443f-82ef-3e8aad554364 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:10.182716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1136/2276] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:10 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:07:10.189903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c801f8-9704-4985-8120-d86636190175 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:10.198451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323f5cd-b15e-4f28-a546-afe6e2e5b1a6 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:10.198746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323f5cd-b15e-4f28-a546-afe6e2e5b1a6 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:10.199033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a323f5cd-b15e-4f28-a546-afe6e2e5b1a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:10.203824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c801f8-9704-4985-8120-d86636190175 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:10.209046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884fbea9-3e44-4a30-8697-ab9c9c52d294 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:10.210047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884fbea9-3e44-4a30-8697-ab9c9c52d294 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:10.210389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884fbea9-3e44-4a30-8697-ab9c9c52d294 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:10.210728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884fbea9-3e44-4a30-8697-ab9c9c52d294 admin admin] PATH_INFO: `/v3/projects/a50ce98768be4aae85de6710cc2a242d/users/5a673b383401416fa256965a9724b4d9/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:10.216977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a323f5cd-b15e-4f28-a546-afe6e2e5b1a6 None None] MFA Rules not processed for user `24343309fb924981a37899f56f15b8ba`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:10.218214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a323f5cd-b15e-4f28-a546-afe6e2e5b1a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:10.226688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884fbea9-3e44-4a30-8697-ab9c9c52d294 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a50ce98768be4aae85de6710cc2a242d, user_id=5a673b383401416fa256965a9724b4d9, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:10.229666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884fbea9-3e44-4a30-8697-ab9c9c52d294 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:10.275484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1137/2277] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:10 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:10.362828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1141/2278] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:07:10 2022] PUT /identity/v3/projects/a50ce98768be4aae85de6710cc2a242d/users/5a673b383401416fa256965a9724b4d9/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:10.367943 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1e50fa-84e9-46ae-9ff0-54db7eb83b38 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:10.368239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1e50fa-84e9-46ae-9ff0-54db7eb83b38 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:10.368524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1e50fa-84e9-46ae-9ff0-54db7eb83b38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:10.404605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4f1e50fa-84e9-46ae-9ff0-54db7eb83b38 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:10.405894 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4f1e50fa-84e9-46ae-9ff0-54db7eb83b38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:10.486625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1138/2279] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:10.496182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884fbea9-3e44-4a30-8697-ab9c9c52d294 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:10.498785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-884fbea9-3e44-4a30-8697-ab9c9c52d294 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:10.551344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1e50fa-84e9-46ae-9ff0-54db7eb83b38 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:10.552664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4f1e50fa-84e9-46ae-9ff0-54db7eb83b38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:10.570459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1e50fa-84e9-46ae-9ff0-54db7eb83b38 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:10.583151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884fbea9-3e44-4a30-8697-ab9c9c52d294 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:10.590704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:10.590704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:10.590704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:10.590704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] PATH_INFO: `/v3/users/74dd023f4a024e8898e1d4b55cc71d60` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:10.591111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] RBAC: Authorizing `identity:delete_user(user_id=74dd023f4a024e8898e1d4b55cc71d60)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:10.591728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4201ffa5-e2f9-452d-85e8-5bd6e00e87e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:10.591728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4201ffa5-e2f9-452d-85e8-5bd6e00e87e1 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:10.592241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4201ffa5-e2f9-452d-85e8-5bd6e00e87e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:10.592523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4201ffa5-e2f9-452d-85e8-5bd6e00e87e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:10.592866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:10.608845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4201ffa5-e2f9-452d-85e8-5bd6e00e87e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:10.611262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4201ffa5-e2f9-452d-85e8-5bd6e00e87e1 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:10.626499 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1142/2280] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:10.884669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '74dd023f4a024e8898e1d4b55cc71d60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:10.885302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '74dd023f4a024e8898e1d4b55cc71d60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:10.892587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '74dd023f4a024e8898e1d4b55cc71d60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:10.957533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '74dd023f4a024e8898e1d4b55cc71d60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:10.958163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '74dd023f4a024e8898e1d4b55cc71d60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:10.990794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1139/2281] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:10 2022] DELETE /identity/v3/users/74dd023f4a024e8898e1d4b55cc71d60 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:11.000582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f1140b-1fa6-43dc-86d9-29a14983eedd None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:11.003814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f1140b-1fa6-43dc-86d9-29a14983eedd None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:11.003814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f1140b-1fa6-43dc-86d9-29a14983eedd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:11.033651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-51f1140b-1fa6-43dc-86d9-29a14983eedd None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:11.035165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-51f1140b-1fa6-43dc-86d9-29a14983eedd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:11.181720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1143/2282] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:11.192160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:11.193648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:11.248455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:11.265180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f80a3ac8-980e-4e26-8367-a478710f6e71 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:11.268759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa046582-f471-417f-9643-9c328859c642 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:11.269462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa046582-f471-417f-9643-9c328859c642 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:11.269749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa046582-f471-417f-9643-9c328859c642 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:11.270028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa046582-f471-417f-9643-9c328859c642 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:11.279007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa046582-f471-417f-9643-9c328859c642 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:11.280435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa046582-f471-417f-9643-9c328859c642 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:11.290553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1140/2283] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:11.943451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f1140b-1fa6-43dc-86d9-29a14983eedd None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:11.946327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-51f1140b-1fa6-43dc-86d9-29a14983eedd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:11.960710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f1140b-1fa6-43dc-86d9-29a14983eedd None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:11.966268 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:11.967349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:11.967638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:11.968106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] PATH_INFO: `/v3/projects/95e50863a8ad44ce84e13b85301e0dc9` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:11.969266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] RBAC: Authorizing `identity:delete_project(project_id=95e50863a8ad44ce84e13b85301e0dc9)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:11.971895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.203754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95e50863a8ad44ce84e13b85301e0dc9', 'request_id': 'req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:12.306360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95e50863a8ad44ce84e13b85301e0dc9', 'request_id': 'req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:12.313795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95e50863a8ad44ce84e13b85301e0dc9', 'request_id': 'req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:12.324489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] The token cache is being invalidate because project 95e50863a8ad44ce84e13b85301e0dc9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:12.324797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:12.325944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6c169f3a-66ca-48fb-bfca-fdebbbfb15ca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:12.331079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1144/2284] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:11 2022] DELETE /identity/v3/projects/95e50863a8ad44ce84e13b85301e0dc9 => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:12.337183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d3ccff-30b9-4a30-92db-9d518e2cbdf0 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.337636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d3ccff-30b9-4a30-92db-9d518e2cbdf0 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.337818 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d3ccff-30b9-4a30-92db-9d518e2cbdf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.361101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-53d3ccff-30b9-4a30-92db-9d518e2cbdf0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:12.362331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-53d3ccff-30b9-4a30-92db-9d518e2cbdf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:12.432496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1141/2285] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:12.440742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa9d3f1-cac6-44ee-952b-85e33df38f9d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.441150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa9d3f1-cac6-44ee-952b-85e33df38f9d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.441576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa9d3f1-cac6-44ee-952b-85e33df38f9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.464601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca7951-fe16-49ca-8308-c9bbb9fb5f7e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.464969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca7951-fe16-49ca-8308-c9bbb9fb5f7e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.465396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ca7951-fe16-49ca-8308-c9bbb9fb5f7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.484272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-2aa9d3f1-cac6-44ee-952b-85e33df38f9d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:12.487279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:12.488846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2aa9d3f1-cac6-44ee-952b-85e33df38f9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:12.505364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e8ca7951-fe16-49ca-8308-c9bbb9fb5f7e None None] MFA Rules not processed for user `5a673b383401416fa256965a9724b4d9`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:12.506625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e8ca7951-fe16-49ca-8308-c9bbb9fb5f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:12.510422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1145/2286] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:12.516952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa9d3f1-cac6-44ee-952b-85e33df38f9d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.526561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa9d3f1-cac6-44ee-952b-85e33df38f9d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.534326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02664ce6-e2f7-4512-90e8-8cd6f67d76c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.535330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02664ce6-e2f7-4512-90e8-8cd6f67d76c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.535713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02664ce6-e2f7-4512-90e8-8cd6f67d76c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.536132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02664ce6-e2f7-4512-90e8-8cd6f67d76c2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.538634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02664ce6-e2f7-4512-90e8-8cd6f67d76c2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:07:12.539068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02664ce6-e2f7-4512-90e8-8cd6f67d76c2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:12.544105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02664ce6-e2f7-4512-90e8-8cd6f67d76c2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.556302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1146/2287] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:07:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:07:12.562079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02664ce6-e2f7-4512-90e8-8cd6f67d76c2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.579058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02664ce6-e2f7-4512-90e8-8cd6f67d76c2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.583677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e463d2-9204-4666-b425-f5593f1dec12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.584436 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e463d2-9204-4666-b425-f5593f1dec12 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.584814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e463d2-9204-4666-b425-f5593f1dec12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.585174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e463d2-9204-4666-b425-f5593f1dec12 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.585886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e463d2-9204-4666-b425-f5593f1dec12 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:12.587810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e463d2-9204-4666-b425-f5593f1dec12 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.597033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1142/2288] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:12 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:12.603646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca7951-fe16-49ca-8308-c9bbb9fb5f7e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.603646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e8ca7951-fe16-49ca-8308-c9bbb9fb5f7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:12.619211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ca7951-fe16-49ca-8308-c9bbb9fb5f7e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.623717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfa05f6-717e-4d6a-a805-7871e182ba30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.624366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa05f6-717e-4d6a-a805-7871e182ba30 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.624679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa05f6-717e-4d6a-a805-7871e182ba30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.624983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfa05f6-717e-4d6a-a805-7871e182ba30 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.625719 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfa05f6-717e-4d6a-a805-7871e182ba30 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:12.627620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfa05f6-717e-4d6a-a805-7871e182ba30 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.667492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1147/2289] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:07:12 2022] POST /identity/v3/projects => generated 373 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:07:12.672678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e463d2-9204-4666-b425-f5593f1dec12 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.733993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1143/2290] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:07:12 2022] POST /identity/v3/projects => generated 371 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:07:12.736078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfa05f6-717e-4d6a-a805-7871e182ba30 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.737068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e463d2-9204-4666-b425-f5593f1dec12 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.799036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:12.809518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164ccf78-fc4a-47d8-bb5f-2639cf91579a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.810224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164ccf78-fc4a-47d8-bb5f-2639cf91579a admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.810588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164ccf78-fc4a-47d8-bb5f-2639cf91579a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.810784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164ccf78-fc4a-47d8-bb5f-2639cf91579a admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.811469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164ccf78-fc4a-47d8-bb5f-2639cf91579a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:12.813517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164ccf78-fc4a-47d8-bb5f-2639cf91579a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.815399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfa05f6-717e-4d6a-a805-7871e182ba30 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.820251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255d5e5e-7565-4241-a067-b23c49f30f5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.820954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255d5e5e-7565-4241-a067-b23c49f30f5a admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.821247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255d5e5e-7565-4241-a067-b23c49f30f5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.821524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255d5e5e-7565-4241-a067-b23c49f30f5a admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.822238 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255d5e5e-7565-4241-a067-b23c49f30f5a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:12.824168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255d5e5e-7565-4241-a067-b23c49f30f5a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.883887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1148/2291] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:07:12 2022] POST /identity/v3/users => generated 287 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:07:12.889953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164ccf78-fc4a-47d8-bb5f-2639cf91579a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.897235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1144/2292] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:07:12 2022] POST /identity/v3/users => generated 292 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:07:12.900590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255d5e5e-7565-4241-a067-b23c49f30f5a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.901393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164ccf78-fc4a-47d8-bb5f-2639cf91579a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.914335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255d5e5e-7565-4241-a067-b23c49f30f5a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.916307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445e9812-77ff-4515-8b69-8d5fbc415f9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.917096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445e9812-77ff-4515-8b69-8d5fbc415f9c admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.917387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445e9812-77ff-4515-8b69-8d5fbc415f9c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.917667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445e9812-77ff-4515-8b69-8d5fbc415f9c admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.918437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445e9812-77ff-4515-8b69-8d5fbc415f9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:12.919347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dc1469-9274-4e0e-9770-ea0f19c5e2ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.920587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dc1469-9274-4e0e-9770-ea0f19c5e2ce admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.920971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dc1469-9274-4e0e-9770-ea0f19c5e2ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.921087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445e9812-77ff-4515-8b69-8d5fbc415f9c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.921309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12dc1469-9274-4e0e-9770-ea0f19c5e2ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.922232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dc1469-9274-4e0e-9770-ea0f19c5e2ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:12.924705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12dc1469-9274-4e0e-9770-ea0f19c5e2ce admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.932014 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1149/2293] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:12 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:07:12.935745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1145/2294] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:12 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:07:12.935967 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445e9812-77ff-4515-8b69-8d5fbc415f9c admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.939784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dc1469-9274-4e0e-9770-ea0f19c5e2ce admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.947134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445e9812-77ff-4515-8b69-8d5fbc415f9c admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.950023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12dc1469-9274-4e0e-9770-ea0f19c5e2ce admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.952595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd824c4a-908e-43ca-ac0b-57bd4c907612 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.953309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd824c4a-908e-43ca-ac0b-57bd4c907612 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.953594 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd824c4a-908e-43ca-ac0b-57bd4c907612 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.954034 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd824c4a-908e-43ca-ac0b-57bd4c907612 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.954856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd824c4a-908e-43ca-ac0b-57bd4c907612 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:12.956491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd824c4a-908e-43ca-ac0b-57bd4c907612 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.956630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54017a72-8c69-4781-baa2-16e661ef2167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.957412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54017a72-8c69-4781-baa2-16e661ef2167 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.957699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54017a72-8c69-4781-baa2-16e661ef2167 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.958004 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54017a72-8c69-4781-baa2-16e661ef2167 admin admin] PATH_INFO: `/v3/projects/a57ed02c22084f46965ee0df9cf672de/users/4704b2c97dc4421da2e1b03c27707d7c/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:12.972715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1150/2295] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:12 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:07:12.976705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd824c4a-908e-43ca-ac0b-57bd4c907612 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:12.976945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54017a72-8c69-4781-baa2-16e661ef2167 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a57ed02c22084f46965ee0df9cf672de, user_id=4704b2c97dc4421da2e1b03c27707d7c, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:12.979621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54017a72-8c69-4781-baa2-16e661ef2167 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:12.987571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd824c4a-908e-43ca-ac0b-57bd4c907612 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:12.992318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdeaf9ba-f90c-4d24-bbf6-954a3f067896 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:12.993254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdeaf9ba-f90c-4d24-bbf6-954a3f067896 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:12.993542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdeaf9ba-f90c-4d24-bbf6-954a3f067896 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:12.993823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdeaf9ba-f90c-4d24-bbf6-954a3f067896 admin admin] PATH_INFO: `/v3/projects/75273f09637240f3a5cc7129927765f1/users/0a1368d42efc4543a908a61c48ec4472/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:13.011226 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdeaf9ba-f90c-4d24-bbf6-954a3f067896 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75273f09637240f3a5cc7129927765f1, user_id=0a1368d42efc4543a908a61c48ec4472, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:13.014561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdeaf9ba-f90c-4d24-bbf6-954a3f067896 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:13.078108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1146/2296] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:07:12 2022] PUT /identity/v3/projects/a57ed02c22084f46965ee0df9cf672de/users/4704b2c97dc4421da2e1b03c27707d7c/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:13.081941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54017a72-8c69-4781-baa2-16e661ef2167 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:13.108277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1151/2297] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:07:12 2022] PUT /identity/v3/projects/75273f09637240f3a5cc7129927765f1/users/0a1368d42efc4543a908a61c48ec4472/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:07:13.113991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6953e212-80ff-4b9c-9694-df0a6c4f3dc5 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:13.114355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6953e212-80ff-4b9c-9694-df0a6c4f3dc5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:13.114710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6953e212-80ff-4b9c-9694-df0a6c4f3dc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:13.131685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-6953e212-80ff-4b9c-9694-df0a6c4f3dc5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:13.133039 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6953e212-80ff-4b9c-9694-df0a6c4f3dc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:13.156655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54017a72-8c69-4781-baa2-16e661ef2167 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:13.160802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:13.224292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1152/2298] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:13.232635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6953e212-80ff-4b9c-9694-df0a6c4f3dc5 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:13.234164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6953e212-80ff-4b9c-9694-df0a6c4f3dc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:13.236583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206eb9c6-5e7c-4c21-8953-ccf470f887d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:13.238201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206eb9c6-5e7c-4c21-8953-ccf470f887d1 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:13.238494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206eb9c6-5e7c-4c21-8953-ccf470f887d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:13.239099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206eb9c6-5e7c-4c21-8953-ccf470f887d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:13.241943 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206eb9c6-5e7c-4c21-8953-ccf470f887d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:13.245841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206eb9c6-5e7c-4c21-8953-ccf470f887d1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:13.257454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1147/2299] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:13 2022] GET /identity/v3/roles => generated 1420 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:07:13.265763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206eb9c6-5e7c-4c21-8953-ccf470f887d1 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:13.286198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206eb9c6-5e7c-4c21-8953-ccf470f887d1 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:13.292494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce70eaf6-2766-471a-9d76-39ee976b8e13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:13.293427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce70eaf6-2766-471a-9d76-39ee976b8e13 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:13.293711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce70eaf6-2766-471a-9d76-39ee976b8e13 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:13.294090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce70eaf6-2766-471a-9d76-39ee976b8e13 admin admin] PATH_INFO: `/v3/projects/a57ed02c22084f46965ee0df9cf672de/users/4704b2c97dc4421da2e1b03c27707d7c/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:13.303547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce70eaf6-2766-471a-9d76-39ee976b8e13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a57ed02c22084f46965ee0df9cf672de, user_id=4704b2c97dc4421da2e1b03c27707d7c, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:13.303547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce70eaf6-2766-471a-9d76-39ee976b8e13 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:13.304228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:13.311743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1148/2300] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:07:13 2022] PUT /identity/v3/projects/a57ed02c22084f46965ee0df9cf672de/users/4704b2c97dc4421da2e1b03c27707d7c/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:13.314838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6953e212-80ff-4b9c-9694-df0a6c4f3dc5 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:13.375925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0c2874-2bf7-4f57-b453-a4f0d420f1e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:13.376574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0c2874-2bf7-4f57-b453-a4f0d420f1e9 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:13.376888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0c2874-2bf7-4f57-b453-a4f0d420f1e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:13.377164 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0c2874-2bf7-4f57-b453-a4f0d420f1e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:13.442597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0c2874-2bf7-4f57-b453-a4f0d420f1e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:13.444038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0c2874-2bf7-4f57-b453-a4f0d420f1e9 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:13.454351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1153/2301] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:13.609296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:13.814850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:14.323808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:14.677618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:14.839832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ccb14d-4fed-4b1b-9009-04a0612438c1 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:14.840386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ccb14d-4fed-4b1b-9009-04a0612438c1 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:14.840601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ccb14d-4fed-4b1b-9009-04a0612438c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:14.873347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-90ccb14d-4fed-4b1b-9009-04a0612438c1 None None] MFA Rules not processed for user `4704b2c97dc4421da2e1b03c27707d7c`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:14.874669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-90ccb14d-4fed-4b1b-9009-04a0612438c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:14.950543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1149/2302] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:14 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:14.962207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0c2874-2bf7-4f57-b453-a4f0d420f1e9 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:14.963750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4d0c2874-2bf7-4f57-b453-a4f0d420f1e9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:15.034493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0c2874-2bf7-4f57-b453-a4f0d420f1e9 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:15.038526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3ddd99-2ef1-4b16-a49c-16bd7280e126 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:15.039340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3ddd99-2ef1-4b16-a49c-16bd7280e126 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:15.039794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3ddd99-2ef1-4b16-a49c-16bd7280e126 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:15.040277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3ddd99-2ef1-4b16-a49c-16bd7280e126 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:15.102670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:15.104902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3ddd99-2ef1-4b16-a49c-16bd7280e126 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:15.106356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a3ddd99-2ef1-4b16-a49c-16bd7280e126 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:15.116415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1154/2303] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:14 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:15.612930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:16.089472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:16.466852 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:17.015159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:17.087899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ccb14d-4fed-4b1b-9009-04a0612438c1 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:17.101433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ccb14d-4fed-4b1b-9009-04a0612438c1 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:17.117089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d156fb9-6ce9-4505-bb34-338dbf2258e1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:17.117766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d156fb9-6ce9-4505-bb34-338dbf2258e1 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:17.121215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d156fb9-6ce9-4505-bb34-338dbf2258e1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:17.121771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d156fb9-6ce9-4505-bb34-338dbf2258e1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:17.134146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d156fb9-6ce9-4505-bb34-338dbf2258e1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:17.136600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d156fb9-6ce9-4505-bb34-338dbf2258e1 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:17.176550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1150/2304] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:17 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:17.198162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:17.466929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:17.649440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:18.574947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:18.593671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3ddd99-2ef1-4b16-a49c-16bd7280e126 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:18.595509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8a3ddd99-2ef1-4b16-a49c-16bd7280e126 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:18.633529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3ddd99-2ef1-4b16-a49c-16bd7280e126 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:18.643199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:18.645202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:18.645814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:18.647171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] PATH_INFO: `/v3/users/99cb1cc352c144979ba4f12ff9c08522` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:18.649236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] RBAC: Authorizing `identity:delete_user(user_id=99cb1cc352c144979ba4f12ff9c08522)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:18.651683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:18.731735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:18.854770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99cb1cc352c144979ba4f12ff9c08522'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:18.856110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99cb1cc352c144979ba4f12ff9c08522'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:18.857089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99cb1cc352c144979ba4f12ff9c08522'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:18.868040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99cb1cc352c144979ba4f12ff9c08522'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:18.929224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99cb1cc352c144979ba4f12ff9c08522'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:18.947802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1155/2305] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:18 2022] DELETE /identity/v3/users/99cb1cc352c144979ba4f12ff9c08522 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:18.953097 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9adfee-bc38-493d-b275-b19fd7689702 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:18.953391 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9adfee-bc38-493d-b275-b19fd7689702 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:18.953672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9adfee-bc38-493d-b275-b19fd7689702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:18.970955 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-dd9adfee-bc38-493d-b275-b19fd7689702 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:18.972184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-dd9adfee-bc38-493d-b275-b19fd7689702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:19.045894 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1151/2306] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:19.061583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:19.065165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:19.146444 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14684d68-c132-4fb0-b678-f7766d6a1733 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:19.150244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163bb5a3-f3b0-4446-9251-d33a54ba4672 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:19.150938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163bb5a3-f3b0-4446-9251-d33a54ba4672 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:19.151225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163bb5a3-f3b0-4446-9251-d33a54ba4672 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:19.151504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163bb5a3-f3b0-4446-9251-d33a54ba4672 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:19.161796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163bb5a3-f3b0-4446-9251-d33a54ba4672 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:19.163273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163bb5a3-f3b0-4446-9251-d33a54ba4672 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:19.175629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1156/2307] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:19.684385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:20.095433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:20.419044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9adfee-bc38-493d-b275-b19fd7689702 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:20.422514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-dd9adfee-bc38-493d-b275-b19fd7689702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:20.445287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9adfee-bc38-493d-b275-b19fd7689702 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:20.456056 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:20.461102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:20.461563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:20.461965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] PATH_INFO: `/v3/projects/ef929d61f7494b5f81e0c15d2e47233c` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:20.471659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] RBAC: Authorizing `identity:delete_project(project_id=ef929d61f7494b5f81e0c15d2e47233c)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:20.473870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:20.644387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ef929d61f7494b5f81e0c15d2e47233c', 'request_id': 'req-11ddeed2-7366-4474-b169-070abe64386e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:20.655021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ef929d61f7494b5f81e0c15d2e47233c', 'request_id': 'req-11ddeed2-7366-4474-b169-070abe64386e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:20.667503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ef929d61f7494b5f81e0c15d2e47233c', 'request_id': 'req-11ddeed2-7366-4474-b169-070abe64386e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:20.752868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] The token cache is being invalidate because project ef929d61f7494b5f81e0c15d2e47233c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:20.753178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:20.755295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:20.759924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1152/2308] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:20 2022] DELETE /identity/v3/projects/ef929d61f7494b5f81e0c15d2e47233c => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:20.830623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:20.873877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:21.143697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74942347-dbe7-4226-adde-e9b0f74ae71c None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:21.144219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74942347-dbe7-4226-adde-e9b0f74ae71c None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:21.144325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74942347-dbe7-4226-adde-e9b0f74ae71c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:21.194145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-74942347-dbe7-4226-adde-e9b0f74ae71c None None] MFA Rules not processed for user `0a1368d42efc4543a908a61c48ec4472`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:21.196053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-74942347-dbe7-4226-adde-e9b0f74ae71c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:21.198066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:21.261942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:21.297058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:21.297644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1157/2309] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:21 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:21.299144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:21.414138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ddeed2-7366-4474-b169-070abe64386e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:21.419879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68227d25-1d60-4863-a6de-f94705a5f72c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:21.420734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68227d25-1d60-4863-a6de-f94705a5f72c admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:21.421239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68227d25-1d60-4863-a6de-f94705a5f72c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:21.421794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68227d25-1d60-4863-a6de-f94705a5f72c admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:21.422606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68227d25-1d60-4863-a6de-f94705a5f72c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:21.424729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68227d25-1d60-4863-a6de-f94705a5f72c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:21.511904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1153/2310] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:07:21 2022] POST /identity/v3/projects => generated 373 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:07:21.516870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74942347-dbe7-4226-adde-e9b0f74ae71c None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:21.527887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74942347-dbe7-4226-adde-e9b0f74ae71c None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:21.552474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:21.595020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dea7ee-d22f-4d39-af2b-b3775a34eeac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:21.595770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dea7ee-d22f-4d39-af2b-b3775a34eeac admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:21.596066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dea7ee-d22f-4d39-af2b-b3775a34eeac admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:21.596351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dea7ee-d22f-4d39-af2b-b3775a34eeac admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:21.597101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dea7ee-d22f-4d39-af2b-b3775a34eeac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:21.599091 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8dea7ee-d22f-4d39-af2b-b3775a34eeac admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:21.710081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1158/2311] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:07:21 2022] POST /identity/v3/users => generated 293 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:07:21.711870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68227d25-1d60-4863-a6de-f94705a5f72c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:21.723267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68227d25-1d60-4863-a6de-f94705a5f72c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:21.728949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b480412-c676-4c20-ba99-fc48b23cda44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:21.729201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b480412-c676-4c20-ba99-fc48b23cda44 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:21.729553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b480412-c676-4c20-ba99-fc48b23cda44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:21.729885 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b480412-c676-4c20-ba99-fc48b23cda44 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:21.730685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b480412-c676-4c20-ba99-fc48b23cda44 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:21.732351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b480412-c676-4c20-ba99-fc48b23cda44 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:21.742510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1154/2312] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:21 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:07:21.746618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dea7ee-d22f-4d39-af2b-b3775a34eeac admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:21.761067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8dea7ee-d22f-4d39-af2b-b3775a34eeac admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:21.766420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1978947-7fb7-48ec-88e9-880773d2bddd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:21.767319 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1978947-7fb7-48ec-88e9-880773d2bddd admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:21.767603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1978947-7fb7-48ec-88e9-880773d2bddd admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:21.767891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1978947-7fb7-48ec-88e9-880773d2bddd admin admin] PATH_INFO: `/v3/projects/eefd4f68b49a4490b468a288fec0d57e/users/6a2f463538ae433ea7a71b95721e8378/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:21.783459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1978947-7fb7-48ec-88e9-880773d2bddd admin admin] RBAC: Authorizing `identity:create_grant(project_id=eefd4f68b49a4490b468a288fec0d57e, user_id=6a2f463538ae433ea7a71b95721e8378, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:21.786306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1978947-7fb7-48ec-88e9-880773d2bddd admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:21.901338 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1159/2313] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:07:21 2022] PUT /identity/v3/projects/eefd4f68b49a4490b468a288fec0d57e/users/6a2f463538ae433ea7a71b95721e8378/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:21.905256 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b480412-c676-4c20-ba99-fc48b23cda44 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:21.915985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b480412-c676-4c20-ba99-fc48b23cda44 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:21.981817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1cd066-3353-4c96-95e6-c32f9fa2e1f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:21.981817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1cd066-3353-4c96-95e6-c32f9fa2e1f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:21.981817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1cd066-3353-4c96-95e6-c32f9fa2e1f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:21.981817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1cd066-3353-4c96-95e6-c32f9fa2e1f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:21.982980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1cd066-3353-4c96-95e6-c32f9fa2e1f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:21.984029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1cd066-3353-4c96-95e6-c32f9fa2e1f0 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:21.995025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1155/2314] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:21 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:07:21.999329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1978947-7fb7-48ec-88e9-880773d2bddd admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:22.010259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1978947-7fb7-48ec-88e9-880773d2bddd admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:22.015718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f51a80-95d6-4c73-8625-4628f85a212a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:22.016632 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f51a80-95d6-4c73-8625-4628f85a212a admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:22.016960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f51a80-95d6-4c73-8625-4628f85a212a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:22.017238 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f51a80-95d6-4c73-8625-4628f85a212a admin admin] PATH_INFO: `/v3/projects/eefd4f68b49a4490b468a288fec0d57e/users/6a2f463538ae433ea7a71b95721e8378/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:22.019248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f51a80-95d6-4c73-8625-4628f85a212a admin admin] RBAC: Authorizing `identity:create_grant(project_id=eefd4f68b49a4490b468a288fec0d57e, user_id=6a2f463538ae433ea7a71b95721e8378, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:22.022002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f51a80-95d6-4c73-8625-4628f85a212a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:22.039998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1160/2315] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:07:21 2022] PUT /identity/v3/projects/eefd4f68b49a4490b468a288fec0d57e/users/6a2f463538ae433ea7a71b95721e8378/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:24.757766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a5229e-8447-4e83-8698-ee8bc7a522ad None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:24.758292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a5229e-8447-4e83-8698-ee8bc7a522ad None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:24.758404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a5229e-8447-4e83-8698-ee8bc7a522ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:24.797070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-59a5229e-8447-4e83-8698-ee8bc7a522ad None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:24.798633 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-59a5229e-8447-4e83-8698-ee8bc7a522ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:24.969004 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1156/2316] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:07:24 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 212 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:24.983037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f51a80-95d6-4c73-8625-4628f85a212a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:24.985029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a6f51a80-95d6-4c73-8625-4628f85a212a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:25.193493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f51a80-95d6-4c73-8625-4628f85a212a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:25.212200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490f37e5-bb3e-4963-a911-5138ac4a9aca service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:25.217114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490f37e5-bb3e-4963-a911-5138ac4a9aca service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:25.217700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490f37e5-bb3e-4963-a911-5138ac4a9aca service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:25.218302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490f37e5-bb3e-4963-a911-5138ac4a9aca service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:25.231617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490f37e5-bb3e-4963-a911-5138ac4a9aca service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:25.234397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490f37e5-bb3e-4963-a911-5138ac4a9aca service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:25.241049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:25.260132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1161/2317] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:07:24 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 13:07:32.324044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f96f3f-451b-4b59-a5a2-e594011bc117 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:32.324661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f96f3f-451b-4b59-a5a2-e594011bc117 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:32.324661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f96f3f-451b-4b59-a5a2-e594011bc117 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:32.367202 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-24f96f3f-451b-4b59-a5a2-e594011bc117 None None] MFA Rules not processed for user `6a2f463538ae433ea7a71b95721e8378`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:32.369067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-24f96f3f-451b-4b59-a5a2-e594011bc117 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:32.393159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:32.469126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1157/2318] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:32 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:32.520371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490f37e5-bb3e-4963-a911-5138ac4a9aca service swift] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:32.521946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-490f37e5-bb3e-4963-a911-5138ac4a9aca service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:32.621020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490f37e5-bb3e-4963-a911-5138ac4a9aca service swift] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:32.627709 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e800ffd5-e73d-4812-b117-b513668c0b30 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:32.628387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e800ffd5-e73d-4812-b117-b513668c0b30 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:32.628863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e800ffd5-e73d-4812-b117-b513668c0b30 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:32.629165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e800ffd5-e73d-4812-b117-b513668c0b30 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:32.740433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e800ffd5-e73d-4812-b117-b513668c0b30 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:32.744427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e800ffd5-e73d-4812-b117-b513668c0b30 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:32.779962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1162/2319] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:32 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:37.069712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d27816-cc91-41a1-8c07-263c50c26d09 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:37.070934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d27816-cc91-41a1-8c07-263c50c26d09 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:37.071640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d27816-cc91-41a1-8c07-263c50c26d09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:37.114417 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-42d27816-cc91-41a1-8c07-263c50c26d09 None None] MFA Rules not processed for user `0a1368d42efc4543a908a61c48ec4472`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:37.115906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-42d27816-cc91-41a1-8c07-263c50c26d09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:37.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1158/2320] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:37 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:38.628006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e800ffd5-e73d-4812-b117-b513668c0b30 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:38.630119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e800ffd5-e73d-4812-b117-b513668c0b30 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:38.767262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e800ffd5-e73d-4812-b117-b513668c0b30 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:38.773393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:38.774472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:38.774669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:38.775144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] PATH_INFO: `/v3/users/5a673b383401416fa256965a9724b4d9` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:38.776480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a673b383401416fa256965a9724b4d9)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:38.778618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:38.785592 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:39.033850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a673b383401416fa256965a9724b4d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:39.038596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a673b383401416fa256965a9724b4d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:39.039228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a673b383401416fa256965a9724b4d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:39.056724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a673b383401416fa256965a9724b4d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:39.118396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a673b383401416fa256965a9724b4d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:39.156058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1163/2321] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:38 2022] DELETE /identity/v3/users/5a673b383401416fa256965a9724b4d9 => generated 0 bytes in 528 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:39.968220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d27816-cc91-41a1-8c07-263c50c26d09 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:39.986755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d27816-cc91-41a1-8c07-263c50c26d09 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:40.044467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:40.045404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:40.045691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:40.045977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] PATH_INFO: `/v3/projects/a50ce98768be4aae85de6710cc2a242d` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:40.047206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a50ce98768be4aae85de6710cc2a242d)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:40.049301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:40.359213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a50ce98768be4aae85de6710cc2a242d', 'request_id': 'req-7661a864-14a7-40f7-a51f-56005b1a8d16'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.367072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a50ce98768be4aae85de6710cc2a242d', 'request_id': 'req-7661a864-14a7-40f7-a51f-56005b1a8d16'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.384605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a50ce98768be4aae85de6710cc2a242d', 'request_id': 'req-7661a864-14a7-40f7-a51f-56005b1a8d16'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.467421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] The token cache is being invalidate because project a50ce98768be4aae85de6710cc2a242d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:40.467644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.468287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.472800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1159/2322] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:39 2022] DELETE /identity/v3/projects/a50ce98768be4aae85de6710cc2a242d => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:40.474236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:40.478435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:40.492361 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a18543-5dde-4e5d-9ef9-d47c3c81239d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:40.592507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:40.596871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:40.597231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:40.597528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] PATH_INFO: `/v3/users/4704b2c97dc4421da2e1b03c27707d7c` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:40.600184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] RBAC: Authorizing `identity:delete_user(user_id=4704b2c97dc4421da2e1b03c27707d7c)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:40.611178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:40.867849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4704b2c97dc4421da2e1b03c27707d7c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.868600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4704b2c97dc4421da2e1b03c27707d7c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.869253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4704b2c97dc4421da2e1b03c27707d7c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.885786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4704b2c97dc4421da2e1b03c27707d7c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.941403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-5ee3b529-770b-4e87-8eab-211ed03981dc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4704b2c97dc4421da2e1b03c27707d7c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:40.978689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1164/2323] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:40 2022] DELETE /identity/v3/users/4704b2c97dc4421da2e1b03c27707d7c => generated 0 bytes in 499 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:41.298715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:41.299163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:41.319990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7661a864-14a7-40f7-a51f-56005b1a8d16 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:41.365175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:41.393198 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:41.394585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:41.394996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:41.395287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] PATH_INFO: `/v3/projects/a57ed02c22084f46965ee0df9cf672de` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:41.396605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] RBAC: Authorizing `identity:delete_project(project_id=a57ed02c22084f46965ee0df9cf672de)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:41.398627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:41.525746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a57ed02c22084f46965ee0df9cf672de', 'request_id': 'req-ce99f969-4af2-408b-b930-b7174b7ab35e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:41.531930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a57ed02c22084f46965ee0df9cf672de', 'request_id': 'req-ce99f969-4af2-408b-b930-b7174b7ab35e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:41.537877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a57ed02c22084f46965ee0df9cf672de', 'request_id': 'req-ce99f969-4af2-408b-b930-b7174b7ab35e'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:41.615083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] The token cache is being invalidate because project a57ed02c22084f46965ee0df9cf672de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:41.615383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:41.616019 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ce99f969-4af2-408b-b930-b7174b7ab35e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:41.618117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1160/2324] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:41 2022] DELETE /identity/v3/projects/a57ed02c22084f46965ee0df9cf672de => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:41.623544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5239b0-a590-4879-b2ae-0c748ece2b56 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:41.624051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5239b0-a590-4879-b2ae-0c748ece2b56 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:41.624252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5239b0-a590-4879-b2ae-0c748ece2b56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:41.629471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:41.692671 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ae5239b0-a590-4879-b2ae-0c748ece2b56 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:41.694061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ae5239b0-a590-4879-b2ae-0c748ece2b56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:41.755102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1165/2325] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:41.762866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62f49eb-0a76-451a-a465-42927abb8147 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:41.763234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62f49eb-0a76-451a-a465-42927abb8147 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:41.763604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62f49eb-0a76-451a-a465-42927abb8147 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:41.785676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c62f49eb-0a76-451a-a465-42927abb8147 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:41.786811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c62f49eb-0a76-451a-a465-42927abb8147 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:41.795738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1161/2326] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:41.800082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5239b0-a590-4879-b2ae-0c748ece2b56 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:41.810000 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5239b0-a590-4879-b2ae-0c748ece2b56 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:41.817012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce011a23-2806-41ec-926a-aee62d609fd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:41.817685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce011a23-2806-41ec-926a-aee62d609fd4 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:41.818163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce011a23-2806-41ec-926a-aee62d609fd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:41.818442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce011a23-2806-41ec-926a-aee62d609fd4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:41.819180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce011a23-2806-41ec-926a-aee62d609fd4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:07:41.819587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce011a23-2806-41ec-926a-aee62d609fd4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:41.821462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce011a23-2806-41ec-926a-aee62d609fd4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:41.837057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1166/2327] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:07:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:07:41.838411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62f49eb-0a76-451a-a465-42927abb8147 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:41.852684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62f49eb-0a76-451a-a465-42927abb8147 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:41.856304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f217cc43-ae1d-4b86-9842-961771bd3afa None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:41.856599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f217cc43-ae1d-4b86-9842-961771bd3afa None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:41.856916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f217cc43-ae1d-4b86-9842-961771bd3afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:41.858555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34633d9d-ab80-464c-add1-5bced3e87c76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:41.859413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34633d9d-ab80-464c-add1-5bced3e87c76 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:41.859814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34633d9d-ab80-464c-add1-5bced3e87c76 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:41.860185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34633d9d-ab80-464c-add1-5bced3e87c76 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:41.861274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34633d9d-ab80-464c-add1-5bced3e87c76 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:41.863327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34633d9d-ab80-464c-add1-5bced3e87c76 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:41.875940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-f217cc43-ae1d-4b86-9842-961771bd3afa None None] MFA Rules not processed for user `24343309fb924981a37899f56f15b8ba`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:41.878392 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f217cc43-ae1d-4b86-9842-961771bd3afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:41.933079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1167/2328] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:41 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:41.983349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1162/2329] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:07:41 2022] POST /identity/v3/projects => generated 371 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:07:41.988128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f217cc43-ae1d-4b86-9842-961771bd3afa None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:42.009046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f217cc43-ae1d-4b86-9842-961771bd3afa None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:42.065871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44df82f2-d500-4b71-ac86-c7cc88d570b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:42.066828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44df82f2-d500-4b71-ac86-c7cc88d570b5 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:42.067200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44df82f2-d500-4b71-ac86-c7cc88d570b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:42.067728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44df82f2-d500-4b71-ac86-c7cc88d570b5 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:42.069674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44df82f2-d500-4b71-ac86-c7cc88d570b5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:42.072281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44df82f2-d500-4b71-ac86-c7cc88d570b5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:42.084759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:42.253096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1168/2330] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:07:41 2022] POST /identity/v3/users => generated 286 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:07:42.254242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34633d9d-ab80-464c-add1-5bced3e87c76 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:42.265150 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34633d9d-ab80-464c-add1-5bced3e87c76 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:42.271333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba48f71-b53d-4c52-8cfd-35640c2fffc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:42.271333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba48f71-b53d-4c52-8cfd-35640c2fffc1 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:42.271333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba48f71-b53d-4c52-8cfd-35640c2fffc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:42.271692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba48f71-b53d-4c52-8cfd-35640c2fffc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:42.272373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba48f71-b53d-4c52-8cfd-35640c2fffc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:42.281559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba48f71-b53d-4c52-8cfd-35640c2fffc1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:42.295001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1163/2331] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:42 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:07:42.299262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44df82f2-d500-4b71-ac86-c7cc88d570b5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:42.309475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44df82f2-d500-4b71-ac86-c7cc88d570b5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:42.314253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fa10d4-f630-4b28-8f36-7728998874ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:42.315045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fa10d4-f630-4b28-8f36-7728998874ca admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:42.315357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fa10d4-f630-4b28-8f36-7728998874ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:42.315640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fa10d4-f630-4b28-8f36-7728998874ca admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:42.321626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fa10d4-f630-4b28-8f36-7728998874ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:42.321626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fa10d4-f630-4b28-8f36-7728998874ca admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:42.333355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1169/2332] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:07:42 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:07:42.334534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba48f71-b53d-4c52-8cfd-35640c2fffc1 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:42.345927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba48f71-b53d-4c52-8cfd-35640c2fffc1 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:42.351649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f22344e-f288-4bb5-9d33-58fbbaf601df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:42.351827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22344e-f288-4bb5-9d33-58fbbaf601df admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:42.352114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22344e-f288-4bb5-9d33-58fbbaf601df admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:42.352395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f22344e-f288-4bb5-9d33-58fbbaf601df admin admin] PATH_INFO: `/v3/projects/e407ca6178974b41a3bf4dfc320c7774/users/00c5844fd2ba4131b80e1f5c8a80ca7a/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:42.368183 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f22344e-f288-4bb5-9d33-58fbbaf601df admin admin] RBAC: Authorizing `identity:create_grant(project_id=e407ca6178974b41a3bf4dfc320c7774, user_id=00c5844fd2ba4131b80e1f5c8a80ca7a, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:42.370893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f22344e-f288-4bb5-9d33-58fbbaf601df admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:42.478648 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1164/2333] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:07:42 2022] PUT /identity/v3/projects/e407ca6178974b41a3bf4dfc320c7774/users/00c5844fd2ba4131b80e1f5c8a80ca7a/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:07:42.490195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51b6487-5b8e-49ad-996e-14bdba657ba7 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:42.490489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51b6487-5b8e-49ad-996e-14bdba657ba7 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:42.490773 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51b6487-5b8e-49ad-996e-14bdba657ba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:42.517230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-c51b6487-5b8e-49ad-996e-14bdba657ba7 None None] MFA Rules not processed for user `00c5844fd2ba4131b80e1f5c8a80ca7a`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:42.518439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c51b6487-5b8e-49ad-996e-14bdba657ba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:42.590542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1170/2334] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:42 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:42.597671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f22344e-f288-4bb5-9d33-58fbbaf601df admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:42.599791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5f22344e-f288-4bb5-9d33-58fbbaf601df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:42.679141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f22344e-f288-4bb5-9d33-58fbbaf601df admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:42.683956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fe4edb-a8fc-408e-9a95-d8c48c25761a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:42.684684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fe4edb-a8fc-408e-9a95-d8c48c25761a service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:42.685750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fe4edb-a8fc-408e-9a95-d8c48c25761a service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:42.686132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fe4edb-a8fc-408e-9a95-d8c48c25761a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:42.698173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fe4edb-a8fc-408e-9a95-d8c48c25761a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:42.701509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fe4edb-a8fc-408e-9a95-d8c48c25761a service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:42.728640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1165/2335] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:42.760059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51b6487-5b8e-49ad-996e-14bdba657ba7 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:42.829249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51b6487-5b8e-49ad-996e-14bdba657ba7 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:42.846836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:42.848033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:42.848321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:42.848607 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] PATH_INFO: `/v3/users/00c5844fd2ba4131b80e1f5c8a80ca7a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:42.851159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] RBAC: Authorizing `identity:delete_user(user_id=00c5844fd2ba4131b80e1f5c8a80ca7a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:42.853618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:43.072146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '00c5844fd2ba4131b80e1f5c8a80ca7a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:43.072941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '00c5844fd2ba4131b80e1f5c8a80ca7a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:43.073689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '00c5844fd2ba4131b80e1f5c8a80ca7a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:43.100416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '00c5844fd2ba4131b80e1f5c8a80ca7a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:43.331127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '00c5844fd2ba4131b80e1f5c8a80ca7a'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:43.387988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:43.399105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1171/2336] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:07:42 2022] DELETE /identity/v3/users/00c5844fd2ba4131b80e1f5c8a80ca7a => generated 0 bytes in 640 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:43.403782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509f6ab3-480f-498e-96c9-f2f96119e41e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:43.404881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509f6ab3-480f-498e-96c9-f2f96119e41e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:43.405324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509f6ab3-480f-498e-96c9-f2f96119e41e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:43.421897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:43.428469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-509f6ab3-480f-498e-96c9-f2f96119e41e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:43.429723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-509f6ab3-480f-498e-96c9-f2f96119e41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:43.516615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1166/2337] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:07:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:43.527722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:43.530129 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:43.614132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31015145-febf-4a60-8b11-3139e12e5aeb admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:43.618106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f24e15-8447-4d25-b764-ffa854d0fac0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:43.624341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f24e15-8447-4d25-b764-ffa854d0fac0 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:43.624341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f24e15-8447-4d25-b764-ffa854d0fac0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:43.624341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f24e15-8447-4d25-b764-ffa854d0fac0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:43.629654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f24e15-8447-4d25-b764-ffa854d0fac0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:43.631179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2f24e15-8447-4d25-b764-ffa854d0fac0 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:43.642050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1172/2338] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:44.196274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509f6ab3-480f-498e-96c9-f2f96119e41e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:44.198054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-509f6ab3-480f-498e-96c9-f2f96119e41e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:44.221954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509f6ab3-480f-498e-96c9-f2f96119e41e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:44.229449 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:44.230221 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:44.230506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:44.230786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] PATH_INFO: `/v3/projects/e407ca6178974b41a3bf4dfc320c7774` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:44.232012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e407ca6178974b41a3bf4dfc320c7774)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:44.234534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:44.376518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e407ca6178974b41a3bf4dfc320c7774', 'request_id': 'req-6eb38f52-bc6d-446c-9273-f2da8c50ec30'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:44.382745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e407ca6178974b41a3bf4dfc320c7774', 'request_id': 'req-6eb38f52-bc6d-446c-9273-f2da8c50ec30'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:44.388632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e407ca6178974b41a3bf4dfc320c7774', 'request_id': 'req-6eb38f52-bc6d-446c-9273-f2da8c50ec30'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:44.481316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] The token cache is being invalidate because project e407ca6178974b41a3bf4dfc320c7774 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:07:44.481997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:44.484423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6eb38f52-bc6d-446c-9273-f2da8c50ec30 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:07:44.486780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1167/2339] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:07:44 2022] DELETE /identity/v3/projects/e407ca6178974b41a3bf4dfc320c7774 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:07:50.795569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa810cb-1f93-4979-a0de-6a8a6c5c62a9 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:50.796110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa810cb-1f93-4979-a0de-6a8a6c5c62a9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:50.796455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa810cb-1f93-4979-a0de-6a8a6c5c62a9 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:50.798044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1173/2340] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 13:07:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:07:50.805258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389f380e-2e9c-46cc-aa04-d3e96e119040 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:50.805698 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389f380e-2e9c-46cc-aa04-d3e96e119040 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:50.806052 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389f380e-2e9c-46cc-aa04-d3e96e119040 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:50.875077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-389f380e-2e9c-46cc-aa04-d3e96e119040 None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:50.876295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-389f380e-2e9c-46cc-aa04-d3e96e119040 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:50.929908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:07:50.965818 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1168/2341] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 13:07:50 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:50.977993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa810cb-1f93-4979-a0de-6a8a6c5c62a9 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:50.979724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-eaa810cb-1f93-4979-a0de-6a8a6c5c62a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:51.039299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa810cb-1f93-4979-a0de-6a8a6c5c62a9 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:51.042957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d4a0c91-859d-4112-ad4f-a2118e45ed94 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a83b4eb626343f3aa5b1c9f3fc828fd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:51.043602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4a0c91-859d-4112-ad4f-a2118e45ed94 service glance] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:51.043889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4a0c91-859d-4112-ad4f-a2118e45ed94 service glance] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:51.044172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d4a0c91-859d-4112-ad4f-a2118e45ed94 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:51.054428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4a0c91-859d-4112-ad4f-a2118e45ed94 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:51.055874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d4a0c91-859d-4112-ad4f-a2118e45ed94 service glance] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:51.067101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1174/2342] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:50 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:07:51.251788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dadb51-45cd-4b8b-9a85-07d43ae9b65a None None] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:51.252082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dadb51-45cd-4b8b-9a85-07d43ae9b65a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:51.252363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05dadb51-45cd-4b8b-9a85-07d43ae9b65a None None] PATH_INFO: `/` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:51.253807 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1169/2343] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 13:07:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:07:51.258561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6664d0fa-7dd4-46d3-9f75-bf5e65b9ae96 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:51.258853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6664d0fa-7dd4-46d3-9f75-bf5e65b9ae96 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:51.259133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6664d0fa-7dd4-46d3-9f75-bf5e65b9ae96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:51.279748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-6664d0fa-7dd4-46d3-9f75-bf5e65b9ae96 None None] MFA Rules not processed for user `d6794257c90e40bca2c8efd31b0c791a`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:51.281132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6664d0fa-7dd4-46d3-9f75-bf5e65b9ae96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:51.344283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1175/2344] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 13:07:51 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:07:51.355911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dadb51-45cd-4b8b-9a85-07d43ae9b65a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:51.357638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-05dadb51-45cd-4b8b-9a85-07d43ae9b65a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:51.370798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05dadb51-45cd-4b8b-9a85-07d43ae9b65a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:51.374636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031920be-2cde-4cfa-955c-4fef88fb1fa9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:51.375282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031920be-2cde-4cfa-955c-4fef88fb1fa9 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:51.375562 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031920be-2cde-4cfa-955c-4fef88fb1fa9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:51.375843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031920be-2cde-4cfa-955c-4fef88fb1fa9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:51.383122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031920be-2cde-4cfa-955c-4fef88fb1fa9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:51.384548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031920be-2cde-4cfa-955c-4fef88fb1fa9 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:51.393157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1170/2345] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:51 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:07:51.477227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513d30c-5367-47b6-a1da-b443f8eb9cd5 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:51.478393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513d30c-5367-47b6-a1da-b443f8eb9cd5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:51.478393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c513d30c-5367-47b6-a1da-b443f8eb9cd5 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:51.479333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1176/2346] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 13:07:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:07:51.484975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b051bf-b5d2-473f-8958-6e2fe3fa9ef8 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:51.485187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b051bf-b5d2-473f-8958-6e2fe3fa9ef8 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:51.485463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b051bf-b5d2-473f-8958-6e2fe3fa9ef8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:51.503195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-40b051bf-b5d2-473f-8958-6e2fe3fa9ef8 None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:07:51.504370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-40b051bf-b5d2-473f-8958-6e2fe3fa9ef8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:51.513985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1171/2347] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 13:07:51 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:07:51.525200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c513d30c-5367-47b6-a1da-b443f8eb9cd5 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:07:51.526877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c513d30c-5367-47b6-a1da-b443f8eb9cd5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:07:51.542116 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c513d30c-5367-47b6-a1da-b443f8eb9cd5 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:07:51.546098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ac89f6-d212-4bd0-9181-5190a98070fe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:07:51.546757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ac89f6-d212-4bd0-9181-5190a98070fe service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:07:51.547044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ac89f6-d212-4bd0-9181-5190a98070fe service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:07:51.547322 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ac89f6-d212-4bd0-9181-5190a98070fe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:07:51.561100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ac89f6-d212-4bd0-9181-5190a98070fe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:07:51.562790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ac89f6-d212-4bd0-9181-5190a98070fe service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:07:51.574404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1177/2348] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:07:51 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:08:01.498200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=21 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:08:01.570761 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:DELETE=1,keystone:SELECT=92 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:08:02.184136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6363671f-c504-4913-8b42-70654150b4ff None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:02.184617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6363671f-c504-4913-8b42-70654150b4ff None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:02.185071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6363671f-c504-4913-8b42-70654150b4ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:02.204427 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6363671f-c504-4913-8b42-70654150b4ff None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:08:02.205669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6363671f-c504-4913-8b42-70654150b4ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:02.268707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1172/2349] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:08:02 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:08:02.292195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ac89f6-d212-4bd0-9181-5190a98070fe service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:08:02.293626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c9ac89f6-d212-4bd0-9181-5190a98070fe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:02.365175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ac89f6-d212-4bd0-9181-5190a98070fe service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:08:02.370183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2009f26-defb-45d9-a168-64eb312d2ddb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:08:02.370825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2009f26-defb-45d9-a168-64eb312d2ddb service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:02.371108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2009f26-defb-45d9-a168-64eb312d2ddb service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:02.371387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2009f26-defb-45d9-a168-64eb312d2ddb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:02.379420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2009f26-defb-45d9-a168-64eb312d2ddb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:08:02.381051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2009f26-defb-45d9-a168-64eb312d2ddb service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:08:02.394335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1178/2350] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:08:02 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 13:08:12.261978 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=25 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:08:12.394881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=32 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:08:24.007053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6363671f-c504-4913-8b42-70654150b4ff None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:08:24.009565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6363671f-c504-4913-8b42-70654150b4ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:24.163254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6363671f-c504-4913-8b42-70654150b4ff None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:08:24.169803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-365c35cf-0c12-42fe-a9a0-1ee1a8d16063 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:08:24.170537 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365c35cf-0c12-42fe-a9a0-1ee1a8d16063 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:24.170824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365c35cf-0c12-42fe-a9a0-1ee1a8d16063 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:24.171106 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-365c35cf-0c12-42fe-a9a0-1ee1a8d16063 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:24.174135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-365c35cf-0c12-42fe-a9a0-1ee1a8d16063 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:24.187646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365c35cf-0c12-42fe-a9a0-1ee1a8d16063 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:08:24.189361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-365c35cf-0c12-42fe-a9a0-1ee1a8d16063 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:08:24.200435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1173/2351] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:08:24 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:08:34.196104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=41 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:08:36.571900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79263bd3-e556-45f9-9b34-8c6376acac1d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:36.573856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79263bd3-e556-45f9-9b34-8c6376acac1d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:36.573856 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79263bd3-e556-45f9-9b34-8c6376acac1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:36.590605 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-79263bd3-e556-45f9-9b34-8c6376acac1d None None] MFA Rules not processed for user `25f73d0bd4e64057a922d11be46eb04c`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:08:36.591965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-79263bd3-e556-45f9-9b34-8c6376acac1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:36.648038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1179/2352] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:08:36 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:08:37.045403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e1ddcd-6c0e-450c-841d-1046223975c8 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:37.045895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e1ddcd-6c0e-450c-841d-1046223975c8 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:37.046064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4e1ddcd-6c0e-450c-841d-1046223975c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:37.104559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a4e1ddcd-6c0e-450c-841d-1046223975c8 None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:08:37.105931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a4e1ddcd-6c0e-450c-841d-1046223975c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:37.138797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1174/2353] 10.222.0.25 () {58 vars in 967 bytes} [Fri Jun 10 13:08:37 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:08:37.146358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79263bd3-e556-45f9-9b34-8c6376acac1d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:08:37.158036 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8647e30c-b972-4e0a-aa56-e4118ee6b6c4 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:37.158360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8647e30c-b972-4e0a-aa56-e4118ee6b6c4 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:37.158646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8647e30c-b972-4e0a-aa56-e4118ee6b6c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:37.174298 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79263bd3-e556-45f9-9b34-8c6376acac1d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:08:37.178159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-8647e30c-b972-4e0a-aa56-e4118ee6b6c4 None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:08:37.179399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8647e30c-b972-4e0a-aa56-e4118ee6b6c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:37.183282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dee2a14-3250-43ce-bfe4-604fcf1704f5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:08:37.184317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee2a14-3250-43ce-bfe4-604fcf1704f5 service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:37.184705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee2a14-3250-43ce-bfe4-604fcf1704f5 service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:37.185108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dee2a14-3250-43ce-bfe4-604fcf1704f5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:37.194764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1175/2354] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:08:37 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:08:37.195995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dee2a14-3250-43ce-bfe4-604fcf1704f5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:08:37.197695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dee2a14-3250-43ce-bfe4-604fcf1704f5 service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:08:37.203803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8647e30c-b972-4e0a-aa56-e4118ee6b6c4 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:08:37.223531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1180/2355] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:08:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 13:08:37.223814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8647e30c-b972-4e0a-aa56-e4118ee6b6c4 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:08:37.230222 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dee42e1-e218-47ae-8bc0-30abe2bca8de service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:08:37.230975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dee42e1-e218-47ae-8bc0-30abe2bca8de service swift] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:37.231337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dee42e1-e218-47ae-8bc0-30abe2bca8de service swift] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:37.231719 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dee42e1-e218-47ae-8bc0-30abe2bca8de service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:37.254013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dee42e1-e218-47ae-8bc0-30abe2bca8de service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:08:37.257205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dee42e1-e218-47ae-8bc0-30abe2bca8de service swift] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:08:37.271191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1176/2356] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:08:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 68 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jun 10 13:08:43.952921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebebe6dc-02bc-49c2-80b4-9df6f1a7640b None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:43.953751 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebebe6dc-02bc-49c2-80b4-9df6f1a7640b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:43.953751 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebebe6dc-02bc-49c2-80b4-9df6f1a7640b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:43.986845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ebebe6dc-02bc-49c2-80b4-9df6f1a7640b None None] MFA Rules not processed for user `849e36f8653d4d1f9a7fb1df8fb913a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:08:43.988472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ebebe6dc-02bc-49c2-80b4-9df6f1a7640b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:44.103005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1181/2357] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:08:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:08:47.266543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=27 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:08:49.757124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dee42e1-e218-47ae-8bc0-30abe2bca8de service swift] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:08:49.771569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dee42e1-e218-47ae-8bc0-30abe2bca8de service swift] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:08:49.775779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ff4fa0-1cec-4efb-9de6-30ad8fca2c6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:08:49.776424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ff4fa0-1cec-4efb-9de6-30ad8fca2c6b service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:49.776706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ff4fa0-1cec-4efb-9de6-30ad8fca2c6b service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:49.777031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ff4fa0-1cec-4efb-9de6-30ad8fca2c6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:49.778919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d5ff4fa0-1cec-4efb-9de6-30ad8fca2c6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:49.955629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ff4fa0-1cec-4efb-9de6-30ad8fca2c6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:08:49.957272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5ff4fa0-1cec-4efb-9de6-30ad8fca2c6b service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:08:49.969628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1177/2358] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:08:49 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:08:53.683999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf12c27-7e88-4b54-a768-85446286a789 None None] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:53.685405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf12c27-7e88-4b54-a768-85446286a789 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:53.686059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf12c27-7e88-4b54-a768-85446286a789 None None] PATH_INFO: `/` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:53.688012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1182/2359] 10.222.0.25 () {54 vars in 921 bytes} [Fri Jun 10 13:08:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jun 10 13:08:53.694982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7e6002-7b39-46db-908f-bc547063a07e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:53.696796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7e6002-7b39-46db-908f-bc547063a07e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:53.697384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7e6002-7b39-46db-908f-bc547063a07e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:53.718163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-2a7e6002-7b39-46db-908f-bc547063a07e None None] MFA Rules not processed for user `e9a0496199134369a0ef740c099eed59`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:08:53.719845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2a7e6002-7b39-46db-908f-bc547063a07e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:53.730513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1178/2360] 10.222.0.25 () {58 vars in 1018 bytes} [Fri Jun 10 13:08:53 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:08:53.746250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf12c27-7e88-4b54-a768-85446286a789 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:08:53.753182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-acf12c27-7e88-4b54-a768-85446286a789 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:53.779914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf12c27-7e88-4b54-a768-85446286a789 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:08:53.790677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acd8dfb-d8d9-4cab-bad2-94582c471a8b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:08:53.791642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acd8dfb-d8d9-4cab-bad2-94582c471a8b service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:53.792070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acd8dfb-d8d9-4cab-bad2-94582c471a8b service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:53.792520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7acd8dfb-d8d9-4cab-bad2-94582c471a8b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:53.804487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acd8dfb-d8d9-4cab-bad2-94582c471a8b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:08:53.806706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7acd8dfb-d8d9-4cab-bad2-94582c471a8b service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:08:53.827641 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1183/2361] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:08:53 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:08:57.287558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d5c08a-ec0e-4956-a1c8-394698fe444d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:57.288049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d5c08a-ec0e-4956-a1c8-394698fe444d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:57.288147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81d5c08a-ec0e-4956-a1c8-394698fe444d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:57.306687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-81d5c08a-ec0e-4956-a1c8-394698fe444d None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:08:57.307880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-81d5c08a-ec0e-4956-a1c8-394698fe444d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:08:57.316491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1179/2362] 10.222.0.25 () {58 vars in 967 bytes} [Fri Jun 10 13:08:57 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:08:57.329385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acd8dfb-d8d9-4cab-bad2-94582c471a8b service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:08:57.341179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7acd8dfb-d8d9-4cab-bad2-94582c471a8b service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:08:57.346437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22f9f6-ace2-4d9b-920c-3d29d09b5b26 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:08:57.347088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22f9f6-ace2-4d9b-920c-3d29d09b5b26 service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:08:57.347371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22f9f6-ace2-4d9b-920c-3d29d09b5b26 service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:08:57.347652 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb22f9f6-ace2-4d9b-920c-3d29d09b5b26 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:08:57.356286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb22f9f6-ace2-4d9b-920c-3d29d09b5b26 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:08:57.363208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb22f9f6-ace2-4d9b-920c-3d29d09b5b26 service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:08:57.367196 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1184/2363] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:08:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 13:09:00.309526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d5c08a-ec0e-4956-a1c8-394698fe444d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:00.311021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-81d5c08a-ec0e-4956-a1c8-394698fe444d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:00.376717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81d5c08a-ec0e-4956-a1c8-394698fe444d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:00.381310 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:00.382181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:00.382467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:00.382745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] PATH_INFO: `/v3/users/24343309fb924981a37899f56f15b8ba` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:00.383948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=24343309fb924981a37899f56f15b8ba)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:00.385920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:00.521856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '24343309fb924981a37899f56f15b8ba'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:00.522541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '24343309fb924981a37899f56f15b8ba'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:00.529392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '24343309fb924981a37899f56f15b8ba'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:00.587067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '24343309fb924981a37899f56f15b8ba'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:00.587711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-ddaf7a61-0ae6-4048-bb86-fedb044266ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '24343309fb924981a37899f56f15b8ba'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:00.602129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:09:00.606923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1180/2364] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:09:00 2022] DELETE /identity/v3/users/24343309fb924981a37899f56f15b8ba => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:09:00.858388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22f9f6-ace2-4d9b-920c-3d29d09b5b26 service swift] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:00.871257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb22f9f6-ace2-4d9b-920c-3d29d09b5b26 service swift] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:00.929708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:00.930555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:00.930841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:00.931123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] PATH_INFO: `/v3/projects/0fdf6af645ba4a02a6a7278eb1fdff0d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:00.932275 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0fdf6af645ba4a02a6a7278eb1fdff0d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:00.934221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:00.936399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:09:01.051308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0fdf6af645ba4a02a6a7278eb1fdff0d', 'request_id': 'req-0ff2e471-a747-4903-80ba-03ba39eb5841'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:01.131012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0fdf6af645ba4a02a6a7278eb1fdff0d', 'request_id': 'req-0ff2e471-a747-4903-80ba-03ba39eb5841'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:01.137385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0fdf6af645ba4a02a6a7278eb1fdff0d', 'request_id': 'req-0ff2e471-a747-4903-80ba-03ba39eb5841'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:01.147875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] The token cache is being invalidate because project 0fdf6af645ba4a02a6a7278eb1fdff0d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:09:01.149081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:01.149081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0ff2e471-a747-4903-80ba-03ba39eb5841 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:01.150472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1185/2365] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:09:00 2022] DELETE /identity/v3/projects/0fdf6af645ba4a02a6a7278eb1fdff0d => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:09:01.155599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96ad998-5ba5-4f47-9e5b-d15a502a2f0d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:01.155889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96ad998-5ba5-4f47-9e5b-d15a502a2f0d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:01.156171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c96ad998-5ba5-4f47-9e5b-d15a502a2f0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:01.175319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-c96ad998-5ba5-4f47-9e5b-d15a502a2f0d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:09:01.176506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c96ad998-5ba5-4f47-9e5b-d15a502a2f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:01.233621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1181/2366] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:09:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:09:01.242474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a5a763-3e88-4354-ba66-ee78548df4e0 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:01.242891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a5a763-3e88-4354-ba66-ee78548df4e0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:01.243179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a5a763-3e88-4354-ba66-ee78548df4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:01.263355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-14a5a763-3e88-4354-ba66-ee78548df4e0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:09:01.264601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-14a5a763-3e88-4354-ba66-ee78548df4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:01.273241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1186/2367] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:09:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:09:01.277656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96ad998-5ba5-4f47-9e5b-d15a502a2f0d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:01.288132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c96ad998-5ba5-4f47-9e5b-d15a502a2f0d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:01.292998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7715ee75-aba2-4d81-ae96-8287266f9421 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:01.293674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7715ee75-aba2-4d81-ae96-8287266f9421 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:01.293976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7715ee75-aba2-4d81-ae96-8287266f9421 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:01.294235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7715ee75-aba2-4d81-ae96-8287266f9421 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:01.294974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7715ee75-aba2-4d81-ae96-8287266f9421 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:09:01.295282 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7715ee75-aba2-4d81-ae96-8287266f9421 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:01.296963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7715ee75-aba2-4d81-ae96-8287266f9421 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:01.313907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1182/2368] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:09:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:09:01.315673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a5a763-3e88-4354-ba66-ee78548df4e0 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:01.326214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a5a763-3e88-4354-ba66-ee78548df4e0 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:01.331307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d4820f-08b1-447a-8df1-d304b3c58a14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:01.332267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d4820f-08b1-447a-8df1-d304b3c58a14 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:01.332758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d4820f-08b1-447a-8df1-d304b3c58a14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:01.333145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d4820f-08b1-447a-8df1-d304b3c58a14 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:01.333927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d4820f-08b1-447a-8df1-d304b3c58a14 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:01.337320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d4820f-08b1-447a-8df1-d304b3c58a14 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:01.447385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1187/2369] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:09:01 2022] POST /identity/v3/projects => generated 381 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:09:01.451757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7715ee75-aba2-4d81-ae96-8287266f9421 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:01.462176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7715ee75-aba2-4d81-ae96-8287266f9421 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:01.519230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f155259b-fb3e-4858-990e-35f63593397d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:01.519897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f155259b-fb3e-4858-990e-35f63593397d admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:01.520178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f155259b-fb3e-4858-990e-35f63593397d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:01.520450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f155259b-fb3e-4858-990e-35f63593397d admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:01.521206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f155259b-fb3e-4858-990e-35f63593397d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:01.523163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f155259b-fb3e-4858-990e-35f63593397d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:01.646555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1183/2370] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:09:01 2022] POST /identity/v3/users => generated 291 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:09:01.650727 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d4820f-08b1-447a-8df1-d304b3c58a14 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:01.663059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d4820f-08b1-447a-8df1-d304b3c58a14 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:01.668206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd492d-85ff-4e1c-a118-3106c7df724b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:01.668912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd492d-85ff-4e1c-a118-3106c7df724b admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:01.669200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd492d-85ff-4e1c-a118-3106c7df724b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:01.669479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6dd492d-85ff-4e1c-a118-3106c7df724b admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:01.670457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd492d-85ff-4e1c-a118-3106c7df724b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:01.672129 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6dd492d-85ff-4e1c-a118-3106c7df724b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:01.682997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1188/2371] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:09:01.687022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f155259b-fb3e-4858-990e-35f63593397d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:01.697359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f155259b-fb3e-4858-990e-35f63593397d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:01.702358 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6129fb4-a66e-4c63-b430-1c2cd1dc60bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:01.703097 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6129fb4-a66e-4c63-b430-1c2cd1dc60bd admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:01.703381 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6129fb4-a66e-4c63-b430-1c2cd1dc60bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:01.703659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6129fb4-a66e-4c63-b430-1c2cd1dc60bd admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:01.704972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6129fb4-a66e-4c63-b430-1c2cd1dc60bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:01.706674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6129fb4-a66e-4c63-b430-1c2cd1dc60bd admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:01.716985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1184/2372] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:09:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:09:01.721799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd492d-85ff-4e1c-a118-3106c7df724b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:01.733382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6dd492d-85ff-4e1c-a118-3106c7df724b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:01.738372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00a2734-8476-4b6b-8c33-3c17215de6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:01.739246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a2734-8476-4b6b-8c33-3c17215de6b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:01.739535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a2734-8476-4b6b-8c33-3c17215de6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:01.739818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00a2734-8476-4b6b-8c33-3c17215de6b0 admin admin] PATH_INFO: `/v3/projects/872e60bdb526409fb9e83746273a9c9a/users/b714804c89804d02b67e647c12133499/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:01.754630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00a2734-8476-4b6b-8c33-3c17215de6b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=872e60bdb526409fb9e83746273a9c9a, user_id=b714804c89804d02b67e647c12133499, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:01.757658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c00a2734-8476-4b6b-8c33-3c17215de6b0 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:01.863675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1189/2373] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:09:01 2022] PUT /identity/v3/projects/872e60bdb526409fb9e83746273a9c9a/users/b714804c89804d02b67e647c12133499/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:09:01.873512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a8370f-a6a5-477f-a1f6-5bdadc7c06d9 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:01.874208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a8370f-a6a5-477f-a1f6-5bdadc7c06d9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:01.874405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1a8370f-a6a5-477f-a1f6-5bdadc7c06d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:01.902540 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-d1a8370f-a6a5-477f-a1f6-5bdadc7c06d9 None None] MFA Rules not processed for user `b714804c89804d02b67e647c12133499`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:09:01.903858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d1a8370f-a6a5-477f-a1f6-5bdadc7c06d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:01.974331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:09:01.983622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1185/2374] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:09:01 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:09:01.999898 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00a2734-8476-4b6b-8c33-3c17215de6b0 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:02.001673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c00a2734-8476-4b6b-8c33-3c17215de6b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:02.083606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00a2734-8476-4b6b-8c33-3c17215de6b0 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:02.087378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e674c377-6ae8-4d12-980e-9353cb189d8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:02.088025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e674c377-6ae8-4d12-980e-9353cb189d8e service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:02.088315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e674c377-6ae8-4d12-980e-9353cb189d8e service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:02.088599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e674c377-6ae8-4d12-980e-9353cb189d8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:02.097203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e674c377-6ae8-4d12-980e-9353cb189d8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:02.098649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e674c377-6ae8-4d12-980e-9353cb189d8e service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:02.107771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1190/2375] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:09:01 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:09:05.008131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a8370f-a6a5-477f-a1f6-5bdadc7c06d9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:05.069328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1a8370f-a6a5-477f-a1f6-5bdadc7c06d9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:05.073957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:05.074820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:05.075101 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:05.075376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] PATH_INFO: `/v3/users/b714804c89804d02b67e647c12133499` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:05.076555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b714804c89804d02b67e647c12133499)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:05.078552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:05.241350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b714804c89804d02b67e647c12133499'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:05.241925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b714804c89804d02b67e647c12133499'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:05.249382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b714804c89804d02b67e647c12133499'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:05.302783 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b714804c89804d02b67e647c12133499'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:05.303398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b714804c89804d02b67e647c12133499'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:05.322611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1186/2376] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:09:05 2022] DELETE /identity/v3/users/b714804c89804d02b67e647c12133499 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:09:05.327028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001edbe5-e442-4be9-8545-84d3a3c43c10 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:05.327478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001edbe5-e442-4be9-8545-84d3a3c43c10 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:05.327604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001edbe5-e442-4be9-8545-84d3a3c43c10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:05.347152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-001edbe5-e442-4be9-8545-84d3a3c43c10 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:09:05.348365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-001edbe5-e442-4be9-8545-84d3a3c43c10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:05.353456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:09:05.409119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1191/2377] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:09:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:09:05.418379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:05.419770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:05.477396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a22dad0-a281-4c3f-b407-6b04813a25c0 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:05.481127 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae20ca7-ddf9-4378-a457-f506872956f8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:05.481931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae20ca7-ddf9-4378-a457-f506872956f8 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:05.482217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae20ca7-ddf9-4378-a457-f506872956f8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:05.482493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae20ca7-ddf9-4378-a457-f506872956f8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:05.493572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae20ca7-ddf9-4378-a457-f506872956f8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:05.494988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae20ca7-ddf9-4378-a457-f506872956f8 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:05.504983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1187/2378] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:09:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:09:05.915814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001edbe5-e442-4be9-8545-84d3a3c43c10 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:05.917445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-001edbe5-e442-4be9-8545-84d3a3c43c10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:05.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001edbe5-e442-4be9-8545-84d3a3c43c10 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:05.945550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:05.946424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:05.946721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:05.947002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] PATH_INFO: `/v3/projects/872e60bdb526409fb9e83746273a9c9a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:05.948209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] RBAC: Authorizing `identity:delete_project(project_id=872e60bdb526409fb9e83746273a9c9a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:05.950175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:06.063488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '872e60bdb526409fb9e83746273a9c9a', 'request_id': 'req-bac56563-3e53-480d-a84c-5b8bd380009d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:06.125407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '872e60bdb526409fb9e83746273a9c9a', 'request_id': 'req-bac56563-3e53-480d-a84c-5b8bd380009d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:06.132146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '872e60bdb526409fb9e83746273a9c9a', 'request_id': 'req-bac56563-3e53-480d-a84c-5b8bd380009d'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:06.143275 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] The token cache is being invalidate because project 872e60bdb526409fb9e83746273a9c9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:09:06.143559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:06.144184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-bac56563-3e53-480d-a84c-5b8bd380009d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:09:06.145752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1192/2379] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:09:05 2022] DELETE /identity/v3/projects/872e60bdb526409fb9e83746273a9c9a => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:09:06.150911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72de8233-c727-469e-ab20-0ff7edb43331 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.151361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72de8233-c727-469e-ab20-0ff7edb43331 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.151469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72de8233-c727-469e-ab20-0ff7edb43331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.170623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-72de8233-c727-469e-ab20-0ff7edb43331 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:09:06.171877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-72de8233-c727-469e-ab20-0ff7edb43331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:06.184463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:09:06.232980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1188/2380] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:09:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:09:06.241103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085e5c0d-e372-46ad-83fc-48aa1c42878f None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.241390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085e5c0d-e372-46ad-83fc-48aa1c42878f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.241669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085e5c0d-e372-46ad-83fc-48aa1c42878f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.264754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-085e5c0d-e372-46ad-83fc-48aa1c42878f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:09:06.264754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-085e5c0d-e372-46ad-83fc-48aa1c42878f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:06.270214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1193/2381] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:09:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:09:06.276339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72de8233-c727-469e-ab20-0ff7edb43331 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:06.286653 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72de8233-c727-469e-ab20-0ff7edb43331 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:06.292145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f7dc50-51fa-46e4-bc1d-eaaeaee7d679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:06.292835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f7dc50-51fa-46e4-bc1d-eaaeaee7d679 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.293123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f7dc50-51fa-46e4-bc1d-eaaeaee7d679 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.293400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f7dc50-51fa-46e4-bc1d-eaaeaee7d679 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.294122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f7dc50-51fa-46e4-bc1d-eaaeaee7d679 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:09:06.295135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f7dc50-51fa-46e4-bc1d-eaaeaee7d679 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:06.296874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f7dc50-51fa-46e4-bc1d-eaaeaee7d679 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:06.308589 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1189/2382] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:09:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:09:06.313019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085e5c0d-e372-46ad-83fc-48aa1c42878f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:06.322536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085e5c0d-e372-46ad-83fc-48aa1c42878f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:06.327409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04138bc0-1d35-4c97-91fa-c6b222ed60ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:06.328073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04138bc0-1d35-4c97-91fa-c6b222ed60ce admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.328360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04138bc0-1d35-4c97-91fa-c6b222ed60ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.328644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04138bc0-1d35-4c97-91fa-c6b222ed60ce admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.329411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04138bc0-1d35-4c97-91fa-c6b222ed60ce admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:06.331850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04138bc0-1d35-4c97-91fa-c6b222ed60ce admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:06.413302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1194/2383] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:09:06 2022] POST /identity/v3/projects => generated 385 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:09:06.417565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f7dc50-51fa-46e4-bc1d-eaaeaee7d679 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:06.428225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f7dc50-51fa-46e4-bc1d-eaaeaee7d679 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:06.489160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8665c7af-ebb0-4c9a-a8d7-75ee096aef4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:06.489832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8665c7af-ebb0-4c9a-a8d7-75ee096aef4f admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.490118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8665c7af-ebb0-4c9a-a8d7-75ee096aef4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.490396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8665c7af-ebb0-4c9a-a8d7-75ee096aef4f admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.491099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8665c7af-ebb0-4c9a-a8d7-75ee096aef4f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:06.493349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8665c7af-ebb0-4c9a-a8d7-75ee096aef4f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:06.553916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1190/2384] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:09:06 2022] POST /identity/v3/users => generated 293 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:09:06.558161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04138bc0-1d35-4c97-91fa-c6b222ed60ce admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:06.568951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04138bc0-1d35-4c97-91fa-c6b222ed60ce admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:06.573598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a970ca-8f24-49d6-a5a5-2dcfcdda6849 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:06.574252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a970ca-8f24-49d6-a5a5-2dcfcdda6849 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.574539 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a970ca-8f24-49d6-a5a5-2dcfcdda6849 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.574823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a970ca-8f24-49d6-a5a5-2dcfcdda6849 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.576002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a970ca-8f24-49d6-a5a5-2dcfcdda6849 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:06.578957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a970ca-8f24-49d6-a5a5-2dcfcdda6849 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:06.590704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1195/2385] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:09:06 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:09:06.594819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8665c7af-ebb0-4c9a-a8d7-75ee096aef4f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:06.605257 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8665c7af-ebb0-4c9a-a8d7-75ee096aef4f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:06.610311 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63a7cd31-5217-4447-ac59-d25162c53b68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:06.610975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a7cd31-5217-4447-ac59-d25162c53b68 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.611261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a7cd31-5217-4447-ac59-d25162c53b68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.611572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63a7cd31-5217-4447-ac59-d25162c53b68 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.612569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a7cd31-5217-4447-ac59-d25162c53b68 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:06.614205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63a7cd31-5217-4447-ac59-d25162c53b68 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:06.624683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1191/2386] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:09:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:09:06.629330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a970ca-8f24-49d6-a5a5-2dcfcdda6849 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:06.639678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a970ca-8f24-49d6-a5a5-2dcfcdda6849 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:06.645433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1135973b-9b17-4b46-a491-083ca6960bb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:06.646511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135973b-9b17-4b46-a491-083ca6960bb7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.646947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135973b-9b17-4b46-a491-083ca6960bb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.647228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1135973b-9b17-4b46-a491-083ca6960bb7 admin admin] PATH_INFO: `/v3/projects/534db947c7aa4d68ac59cdfd7843f6e8/users/d9eb7d339d224a34908d0fc55e3be73c/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.662962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1135973b-9b17-4b46-a491-083ca6960bb7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=534db947c7aa4d68ac59cdfd7843f6e8, user_id=d9eb7d339d224a34908d0fc55e3be73c, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:06.666081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1135973b-9b17-4b46-a491-083ca6960bb7 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:06.737940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1196/2387] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:09:06 2022] PUT /identity/v3/projects/534db947c7aa4d68ac59cdfd7843f6e8/users/d9eb7d339d224a34908d0fc55e3be73c/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:09:06.810517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3889e7-3190-4825-b9b8-6dd7b3c99ca8 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.810785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3889e7-3190-4825-b9b8-6dd7b3c99ca8 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.811071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3889e7-3190-4825-b9b8-6dd7b3c99ca8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.840064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-1e3889e7-3190-4825-b9b8-6dd7b3c99ca8 None None] MFA Rules not processed for user `d9eb7d339d224a34908d0fc55e3be73c`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:09:06.841300 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1e3889e7-3190-4825-b9b8-6dd7b3c99ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:06.903524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1192/2388] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:09:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:09:06.914123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1135973b-9b17-4b46-a491-083ca6960bb7 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:09:06.915799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1135973b-9b17-4b46-a491-083ca6960bb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:09:06.951054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:09:06.972929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1135973b-9b17-4b46-a491-083ca6960bb7 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:09:06.976544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c467c87-4e5b-407e-8e9f-14c6371139e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:09:06.977228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c467c87-4e5b-407e-8e9f-14c6371139e5 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:09:06.977518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c467c87-4e5b-407e-8e9f-14c6371139e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:09:06.977797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c467c87-4e5b-407e-8e9f-14c6371139e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:09:06.986141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c467c87-4e5b-407e-8e9f-14c6371139e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:09:06.988492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c467c87-4e5b-407e-8e9f-14c6371139e5 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:09:06.998088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1197/2389] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:09:06 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:09:16.900654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:09:17.028035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=12 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:10:03.285034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3889e7-3190-4825-b9b8-6dd7b3c99ca8 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:10:03.286502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1e3889e7-3190-4825-b9b8-6dd7b3c99ca8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:03.343214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3889e7-3190-4825-b9b8-6dd7b3c99ca8 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:10:03.346746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf47f05-52f6-4e5a-bb9a-aae9ffa80bd7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:10:03.347388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf47f05-52f6-4e5a-bb9a-aae9ffa80bd7 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:10:03.347672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf47f05-52f6-4e5a-bb9a-aae9ffa80bd7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:10:03.347950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf47f05-52f6-4e5a-bb9a-aae9ffa80bd7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:10:03.349750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0cf47f05-52f6-4e5a-bb9a-aae9ffa80bd7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:03.419028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf47f05-52f6-4e5a-bb9a-aae9ffa80bd7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:10:03.420387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf47f05-52f6-4e5a-bb9a-aae9ffa80bd7 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:10:03.431047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1193/2390] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:10:03 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:10:13.073919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c467c87-4e5b-407e-8e9f-14c6371139e5 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:10:13.075679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3c467c87-4e5b-407e-8e9f-14c6371139e5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:13.091891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c467c87-4e5b-407e-8e9f-14c6371139e5 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:10:13.097346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:10:13.097986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:10:13.098275 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:10:13.098558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:10:13.100267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:13.158359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:10:13.159962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:10:13.172717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1198/2391] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:10:13 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:10:13.426316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=54 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:10:14.594005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf47f05-52f6-4e5a-bb9a-aae9ffa80bd7 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:10:14.603486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf47f05-52f6-4e5a-bb9a-aae9ffa80bd7 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:10:14.607115 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e55077-0dac-46be-b07b-81d9cc93af01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:10:14.607754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e55077-0dac-46be-b07b-81d9cc93af01 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:10:14.608039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e55077-0dac-46be-b07b-81d9cc93af01 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:10:14.608317 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e55077-0dac-46be-b07b-81d9cc93af01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:10:14.610117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-09e55077-0dac-46be-b07b-81d9cc93af01 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:14.623384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e55077-0dac-46be-b07b-81d9cc93af01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:10:14.624833 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e55077-0dac-46be-b07b-81d9cc93af01 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:10:14.636042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1194/2392] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:10:14 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:10:17.662969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:10:17.664440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:17.679373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d5f1b3-de1c-4efc-ac7e-637e7d16c87d service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:10:17.685451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:10:17.686122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:10:17.686408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:10:17.686688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:10:17.688493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:17.746105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:10:17.747702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:10:17.756666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1199/2393] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:10:17 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:10:21.025944 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6038544-2be5-49c9-b259-c683d6a3c8d3 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:10:21.026415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6038544-2be5-49c9-b259-c683d6a3c8d3 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:10:21.026524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6038544-2be5-49c9-b259-c683d6a3c8d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:10:21.044900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-e6038544-2be5-49c9-b259-c683d6a3c8d3 None None] MFA Rules not processed for user `9c8b2c27b69a4776ab6b7b7820da02d9`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:10:21.046009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e6038544-2be5-49c9-b259-c683d6a3c8d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:21.052679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1195/2394] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:10:21 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:10:21.714447 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:10:21.715830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:21.788888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14bf787-c210-4a68-81d9-86fef11e8596 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:10:21.794087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769a5f31-4482-481b-9fca-01f5d79ffb8b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:10:21.794753 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769a5f31-4482-481b-9fca-01f5d79ffb8b service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:10:21.795027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769a5f31-4482-481b-9fca-01f5d79ffb8b service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:10:21.795313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-769a5f31-4482-481b-9fca-01f5d79ffb8b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:10:21.797090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-769a5f31-4482-481b-9fca-01f5d79ffb8b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:21.809097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769a5f31-4482-481b-9fca-01f5d79ffb8b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:10:21.810733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-769a5f31-4482-481b-9fca-01f5d79ffb8b service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:10:21.820853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1200/2395] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:10:21 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:10:24.834729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6038544-2be5-49c9-b259-c683d6a3c8d3 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:10:24.847269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6038544-2be5-49c9-b259-c683d6a3c8d3 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:10:24.851057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b073b1f4-2af7-4e44-8dcd-df5e23656ecc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:10:24.851781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b073b1f4-2af7-4e44-8dcd-df5e23656ecc service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:10:24.852073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b073b1f4-2af7-4e44-8dcd-df5e23656ecc service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:10:24.852375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b073b1f4-2af7-4e44-8dcd-df5e23656ecc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:10:24.854243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b073b1f4-2af7-4e44-8dcd-df5e23656ecc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:24.864279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b073b1f4-2af7-4e44-8dcd-df5e23656ecc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:10:24.865793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b073b1f4-2af7-4e44-8dcd-df5e23656ecc service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:10:24.874223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1196/2396] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:10:24 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:10:25.619454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769a5f31-4482-481b-9fca-01f5d79ffb8b service placement] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:10:25.636711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-769a5f31-4482-481b-9fca-01f5d79ffb8b service placement] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:10:25.642897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:10:25.643650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:10:25.644051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:10:25.644418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:10:25.646359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:10:25.664643 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:10:25.667054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:10:25.678979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1201/2397] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:10:25 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:10:34.870633 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=15 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:10:35.674513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=94 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:11:20.077394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b073b1f4-2af7-4e44-8dcd-df5e23656ecc service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:11:20.090448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b073b1f4-2af7-4e44-8dcd-df5e23656ecc service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:11:20.095890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaeae11-8bdf-4392-8765-bb60fcc15a04 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:11:20.107615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaeae11-8bdf-4392-8765-bb60fcc15a04 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:11:20.107615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaeae11-8bdf-4392-8765-bb60fcc15a04 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:11:20.107615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaeae11-8bdf-4392-8765-bb60fcc15a04 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:11:20.107615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-faaeae11-8bdf-4392-8765-bb60fcc15a04 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:11:20.111086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaeae11-8bdf-4392-8765-bb60fcc15a04 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:11:20.113419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaeae11-8bdf-4392-8765-bb60fcc15a04 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:11:20.124332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1197/2398] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:11:20 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:11:30.119385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=6 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:12:14.551896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:12:14.553359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:12:14.564530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4590e5a4-c958-4a2b-8bfd-8ba828688bd8 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:12:14.568862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24063e0b-7006-4d35-a5bc-4372d68c1b5d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:12:14.569574 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24063e0b-7006-4d35-a5bc-4372d68c1b5d service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:12:14.569858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24063e0b-7006-4d35-a5bc-4372d68c1b5d service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:12:14.570138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24063e0b-7006-4d35-a5bc-4372d68c1b5d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:12:14.571865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-24063e0b-7006-4d35-a5bc-4372d68c1b5d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:12:14.629954 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24063e0b-7006-4d35-a5bc-4372d68c1b5d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:12:14.631904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24063e0b-7006-4d35-a5bc-4372d68c1b5d service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:12:14.640706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1202/2399] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:12:14 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:12:14.705653 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaeae11-8bdf-4392-8765-bb60fcc15a04 service nova] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:12:14.714677 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaeae11-8bdf-4392-8765-bb60fcc15a04 service nova] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:12:14.718238 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:12:14.718865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:12:14.719149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:12:14.719430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:12:14.721200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:12:14.778126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:12:14.779560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:12:14.788332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1198/2400] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:12:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:12:24.636613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=28 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:12:24.784274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=28 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:12:31.980599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24063e0b-7006-4d35-a5bc-4372d68c1b5d service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:12:32.000260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24063e0b-7006-4d35-a5bc-4372d68c1b5d service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:12:32.005663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf79d60-ee06-4406-a803-f26117b1c618 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:12:32.006328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf79d60-ee06-4406-a803-f26117b1c618 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:12:32.006614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf79d60-ee06-4406-a803-f26117b1c618 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:12:32.006894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf79d60-ee06-4406-a803-f26117b1c618 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:12:32.008687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4bf79d60-ee06-4406-a803-f26117b1c618 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:12:32.066827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf79d60-ee06-4406-a803-f26117b1c618 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:12:32.068371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf79d60-ee06-4406-a803-f26117b1c618 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:12:32.077191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1203/2401] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:12:31 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:12:42.073125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=28 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:14:10.590544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:14:10.591868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:14:10.606328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47cf6ae-ff9f-425c-a503-b6cb271b58b1 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:14:10.611438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be28eaf4-5c22-4159-b889-3d61458cbd74 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:14:10.612342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28eaf4-5c22-4159-b889-3d61458cbd74 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:14:10.612626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28eaf4-5c22-4159-b889-3d61458cbd74 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:14:10.612953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28eaf4-5c22-4159-b889-3d61458cbd74 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:14:10.614689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-be28eaf4-5c22-4159-b889-3d61458cbd74 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:14:10.626152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28eaf4-5c22-4159-b889-3d61458cbd74 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:14:10.627740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28eaf4-5c22-4159-b889-3d61458cbd74 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:14:10.638246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1199/2402] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:14:10 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:14:14.877227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf79d60-ee06-4406-a803-f26117b1c618 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:14:14.886968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf79d60-ee06-4406-a803-f26117b1c618 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:14:14.890781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cec4e6-acfe-4271-8566-18befed39c41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:14:14.891415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cec4e6-acfe-4271-8566-18befed39c41 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:14:14.891699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cec4e6-acfe-4271-8566-18befed39c41 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:14:14.891977 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cec4e6-acfe-4271-8566-18befed39c41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:14:14.900241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cec4e6-acfe-4271-8566-18befed39c41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:14:14.901701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28cec4e6-acfe-4271-8566-18befed39c41 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:14:14.910588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1204/2403] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:14:14 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:14:20.633961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=6 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:14:24.906636 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:15:03.688212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be28eaf4-5c22-4159-b889-3d61458cbd74 service placement] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:15:03.697454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be28eaf4-5c22-4159-b889-3d61458cbd74 service placement] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:15:03.701519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdd1685-9a7f-47bc-8d86-8817a1eb78ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:15:03.702144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdd1685-9a7f-47bc-8d86-8817a1eb78ee service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:15:03.702425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdd1685-9a7f-47bc-8d86-8817a1eb78ee service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:15:03.702706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdd1685-9a7f-47bc-8d86-8817a1eb78ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:15:03.712812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdd1685-9a7f-47bc-8d86-8817a1eb78ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:15:03.714229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdd1685-9a7f-47bc-8d86-8817a1eb78ee service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:15:03.725405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1200/2404] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:15:03 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:15:13.663025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cec4e6-acfe-4271-8566-18befed39c41 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:15:13.675651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cec4e6-acfe-4271-8566-18befed39c41 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:15:13.681129 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322ef3d6-7c29-4224-9c0d-f808b4fc75bd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:15:13.681805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322ef3d6-7c29-4224-9c0d-f808b4fc75bd service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:15:13.682090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322ef3d6-7c29-4224-9c0d-f808b4fc75bd service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:15:13.682371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322ef3d6-7c29-4224-9c0d-f808b4fc75bd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:15:13.690786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322ef3d6-7c29-4224-9c0d-f808b4fc75bd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:15:13.692370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322ef3d6-7c29-4224-9c0d-f808b4fc75bd service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:15:13.720624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=6 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:15:13.721583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1205/2405] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:15:13 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:15:17.537436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdd1685-9a7f-47bc-8d86-8817a1eb78ee service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:15:17.546473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdd1685-9a7f-47bc-8d86-8817a1eb78ee service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:15:17.550046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829b26a5-2b48-4c20-a354-5a15d2549cb9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:15:17.550671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829b26a5-2b48-4c20-a354-5a15d2549cb9 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:15:17.550956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829b26a5-2b48-4c20-a354-5a15d2549cb9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:15:17.551234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829b26a5-2b48-4c20-a354-5a15d2549cb9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:15:17.560348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829b26a5-2b48-4c20-a354-5a15d2549cb9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:15:17.562033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829b26a5-2b48-4c20-a354-5a15d2549cb9 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:15:17.588715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1201/2406] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:15:17 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:15:22.432205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322ef3d6-7c29-4224-9c0d-f808b4fc75bd service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:15:22.434287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-322ef3d6-7c29-4224-9c0d-f808b4fc75bd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:15:22.446330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322ef3d6-7c29-4224-9c0d-f808b4fc75bd service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:15:22.449931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f57affb-0eca-47bc-b4a7-0ebbb70bbd73 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:15:22.450565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f57affb-0eca-47bc-b4a7-0ebbb70bbd73 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:15:22.450850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f57affb-0eca-47bc-b4a7-0ebbb70bbd73 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:15:22.451129 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f57affb-0eca-47bc-b4a7-0ebbb70bbd73 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:15:22.461178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f57affb-0eca-47bc-b4a7-0ebbb70bbd73 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:15:22.462591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f57affb-0eca-47bc-b4a7-0ebbb70bbd73 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:15:22.473297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1206/2407] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:15:22 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:15:27.586187 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=13 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:15:32.468538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=19 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:16:17.210469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829b26a5-2b48-4c20-a354-5a15d2549cb9 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:16:17.223614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829b26a5-2b48-4c20-a354-5a15d2549cb9 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:16:17.233343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d943a7-e97a-4692-aaff-ed436e0eed28 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:16:17.234059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d943a7-e97a-4692-aaff-ed436e0eed28 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:16:17.234344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d943a7-e97a-4692-aaff-ed436e0eed28 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:16:17.234890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d943a7-e97a-4692-aaff-ed436e0eed28 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:16:17.243332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d943a7-e97a-4692-aaff-ed436e0eed28 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:16:17.244969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d943a7-e97a-4692-aaff-ed436e0eed28 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:16:17.255260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1202/2408] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:16:17 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:16:27.250437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=6 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:17:05.698699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f57affb-0eca-47bc-b4a7-0ebbb70bbd73 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:17:05.710278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f57affb-0eca-47bc-b4a7-0ebbb70bbd73 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:17:05.716412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f299a4c-3ad7-4aee-9ea2-5714961509b7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:17:05.717095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f299a4c-3ad7-4aee-9ea2-5714961509b7 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:17:05.717380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f299a4c-3ad7-4aee-9ea2-5714961509b7 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:17:05.717660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f299a4c-3ad7-4aee-9ea2-5714961509b7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:17:05.727003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f299a4c-3ad7-4aee-9ea2-5714961509b7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:17:05.728602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f299a4c-3ad7-4aee-9ea2-5714961509b7 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:17:05.740090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1207/2409] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:17:05 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:17:14.242329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d943a7-e97a-4692-aaff-ed436e0eed28 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:17:14.255547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d943a7-e97a-4692-aaff-ed436e0eed28 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:17:14.260789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7b8396-f1bb-49fd-8b3b-fa07c8b4b5ab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:17:14.261429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7b8396-f1bb-49fd-8b3b-fa07c8b4b5ab service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:17:14.261709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7b8396-f1bb-49fd-8b3b-fa07c8b4b5ab service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:17:14.261982 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f7b8396-f1bb-49fd-8b3b-fa07c8b4b5ab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:17:14.270432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7b8396-f1bb-49fd-8b3b-fa07c8b4b5ab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:17:14.272022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f7b8396-f1bb-49fd-8b3b-fa07c8b4b5ab service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:17:14.298169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1203/2410] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:17:14 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:17:15.323996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f299a4c-3ad7-4aee-9ea2-5714961509b7 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:17:15.336476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f299a4c-3ad7-4aee-9ea2-5714961509b7 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:17:15.341805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4224cd74-1cc1-4fce-af2e-d7c9c37f94df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:17:15.342439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4224cd74-1cc1-4fce-af2e-d7c9c37f94df service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:17:15.342723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4224cd74-1cc1-4fce-af2e-d7c9c37f94df service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:17:15.343001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4224cd74-1cc1-4fce-af2e-d7c9c37f94df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:17:15.350901 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4224cd74-1cc1-4fce-af2e-d7c9c37f94df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:17:15.352479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4224cd74-1cc1-4fce-af2e-d7c9c37f94df service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:17:15.381147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1208/2411] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:17:15 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:17:24.295590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=13 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:17:25.378470 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=19 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:18:15.574851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7b8396-f1bb-49fd-8b3b-fa07c8b4b5ab service nova] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:18:15.669839 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f7b8396-f1bb-49fd-8b3b-fa07c8b4b5ab service nova] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:18:15.676298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a061f355-584e-4eaa-90b7-c5fc15abbb66 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:18:15.677099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a061f355-584e-4eaa-90b7-c5fc15abbb66 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:18:15.677351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a061f355-584e-4eaa-90b7-c5fc15abbb66 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:18:15.682045 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a061f355-584e-4eaa-90b7-c5fc15abbb66 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:18:15.687247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a061f355-584e-4eaa-90b7-c5fc15abbb66 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:18:15.688919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a061f355-584e-4eaa-90b7-c5fc15abbb66 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:18:15.717212 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1204/2412] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:18:15 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:18:25.714367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=37 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:19:15.555652 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4224cd74-1cc1-4fce-af2e-d7c9c37f94df service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:19:15.606078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4224cd74-1cc1-4fce-af2e-d7c9c37f94df service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:19:15.611055 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f48100-bdcd-4fce-97fc-27451523e152 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:19:15.611687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f48100-bdcd-4fce-97fc-27451523e152 service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:19:15.611969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f48100-bdcd-4fce-97fc-27451523e152 service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:19:15.612246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4f48100-bdcd-4fce-97fc-27451523e152 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:19:15.620562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f48100-bdcd-4fce-97fc-27451523e152 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:19:15.622166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4f48100-bdcd-4fce-97fc-27451523e152 service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:19:15.631806 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1209/2413] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:19:15 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:19:18.282938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a061f355-584e-4eaa-90b7-c5fc15abbb66 service nova] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:19:18.284554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a061f355-584e-4eaa-90b7-c5fc15abbb66 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:19:18.370096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a061f355-584e-4eaa-90b7-c5fc15abbb66 service nova] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:19:18.373712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0893b8-3a8b-45f1-b36e-b4d726703f4c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:19:18.374354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0893b8-3a8b-45f1-b36e-b4d726703f4c service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:19:18.374641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0893b8-3a8b-45f1-b36e-b4d726703f4c service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:19:18.374920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0893b8-3a8b-45f1-b36e-b4d726703f4c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:19:18.377333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-fa0893b8-3a8b-45f1-b36e-b4d726703f4c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:19:18.455880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0893b8-3a8b-45f1-b36e-b4d726703f4c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:19:18.457370 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0893b8-3a8b-45f1-b36e-b4d726703f4c service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:19:18.474971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1205/2414] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:19:18 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 193 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:19:25.627753 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=24 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:19:28.472347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=74 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:20:15.731631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f48100-bdcd-4fce-97fc-27451523e152 service placement] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:20:15.741037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4f48100-bdcd-4fce-97fc-27451523e152 service placement] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:20:15.744662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e86828d-bb2d-4144-b6fc-97f865adc832 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:20:15.753306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e86828d-bb2d-4144-b6fc-97f865adc832 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:20:15.753306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e86828d-bb2d-4144-b6fc-97f865adc832 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:20:15.753306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e86828d-bb2d-4144-b6fc-97f865adc832 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:20:15.753306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8e86828d-bb2d-4144-b6fc-97f865adc832 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:20:15.841100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e86828d-bb2d-4144-b6fc-97f865adc832 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:20:15.842609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e86828d-bb2d-4144-b6fc-97f865adc832 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:20:15.862853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1210/2415] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:20:15 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:20:15.981109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0893b8-3a8b-45f1-b36e-b4d726703f4c service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:20:15.996375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0893b8-3a8b-45f1-b36e-b4d726703f4c service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:20:15.999851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7df1e90-3162-4cd9-a6e3-f92af3695a8b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:20:15.999851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7df1e90-3162-4cd9-a6e3-f92af3695a8b service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:20:15.999851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7df1e90-3162-4cd9-a6e3-f92af3695a8b service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:20:16.001321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7df1e90-3162-4cd9-a6e3-f92af3695a8b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:20:16.004816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a7df1e90-3162-4cd9-a6e3-f92af3695a8b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:20:16.086977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7df1e90-3162-4cd9-a6e3-f92af3695a8b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:20:16.088484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7df1e90-3162-4cd9-a6e3-f92af3695a8b service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:20:16.098518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1206/2416] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:20:15 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:20:23.169603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e86828d-bb2d-4144-b6fc-97f865adc832 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:20:23.231677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e86828d-bb2d-4144-b6fc-97f865adc832 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:20:23.235280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:20:23.235914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:20:23.236196 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:20:23.236476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:20:23.238893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:20:23.252280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:20:23.259929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:20:23.265434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1211/2417] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:20:23 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:20:26.097134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=33 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:20:33.261022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=69 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:21:18.702496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7df1e90-3162-4cd9-a6e3-f92af3695a8b service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:21:18.711771 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7df1e90-3162-4cd9-a6e3-f92af3695a8b service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:21:18.715546 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:21:18.716305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:21:18.716597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:21:18.716918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:21:18.718844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:21:18.790984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:21:18.792536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:21:18.802628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1207/2418] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:21:18 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:21:21.184647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:21:21.186312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:21:21.210432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11cd4a4-ee68-41ec-8319-43bf16baaf5b service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:21:21.214120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:21:21.216546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:21:21.216546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:21:21.216546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:21:21.217179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:21:21.226536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:21:21.228237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:21:21.236194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1212/2419] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:21:21 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:21:28.798189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=33 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:21:31.232590 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=11 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:22:13.921566 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:22:13.923153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:22:13.938289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116868e2-ed99-4fdf-b4b6-24e77d4850b2 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:22:13.943766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d897cd-3804-46c0-99a7-6e1d002218e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:22:13.944429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d897cd-3804-46c0-99a7-6e1d002218e0 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:22:13.944743 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d897cd-3804-46c0-99a7-6e1d002218e0 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:22:13.945046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d897cd-3804-46c0-99a7-6e1d002218e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:22:13.952789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d897cd-3804-46c0-99a7-6e1d002218e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:22:13.954399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d897cd-3804-46c0-99a7-6e1d002218e0 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:22:13.963596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1208/2420] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:22:13 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:22:16.346541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:22:16.348239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:22:16.359871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9e96c7-39bf-4bae-8fa4-19768ebc42fc service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:22:16.364032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006c2cdf-2b85-43a1-9ff2-b507652032ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:22:16.364819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c2cdf-2b85-43a1-9ff2-b507652032ac service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:22:16.365200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c2cdf-2b85-43a1-9ff2-b507652032ac service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:22:16.365482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c2cdf-2b85-43a1-9ff2-b507652032ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:22:16.367367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-006c2cdf-2b85-43a1-9ff2-b507652032ac service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:22:16.445462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006c2cdf-2b85-43a1-9ff2-b507652032ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:22:16.446914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-006c2cdf-2b85-43a1-9ff2-b507652032ac service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:22:16.456258 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1213/2421] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:22:16 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:22:17.880753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d897cd-3804-46c0-99a7-6e1d002218e0 service nova] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:22:17.892088 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d897cd-3804-46c0-99a7-6e1d002218e0 service nova] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:22:17.898346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7d2618-40bf-45a0-a1cb-6f4d489db2b2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:22:17.898999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7d2618-40bf-45a0-a1cb-6f4d489db2b2 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:22:17.899290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7d2618-40bf-45a0-a1cb-6f4d489db2b2 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:22:17.899572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7d2618-40bf-45a0-a1cb-6f4d489db2b2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:22:17.901531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7b7d2618-40bf-45a0-a1cb-6f4d489db2b2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:22:17.914666 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7d2618-40bf-45a0-a1cb-6f4d489db2b2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:22:17.916269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7d2618-40bf-45a0-a1cb-6f4d489db2b2 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:22:17.927480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1209/2422] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:22:17 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:22:26.451770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=33 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:22:27.922684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=12 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:23:06.706258 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006c2cdf-2b85-43a1-9ff2-b507652032ac service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:23:06.715775 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-006c2cdf-2b85-43a1-9ff2-b507652032ac service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:23:06.719366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3294f4-de86-47a0-8621-cfee46ac293e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:23:06.720017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3294f4-de86-47a0-8621-cfee46ac293e service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:23:06.720306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3294f4-de86-47a0-8621-cfee46ac293e service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:23:06.721125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3294f4-de86-47a0-8621-cfee46ac293e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:23:06.722938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1a3294f4-de86-47a0-8621-cfee46ac293e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:23:06.810152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3294f4-de86-47a0-8621-cfee46ac293e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:23:06.811623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3294f4-de86-47a0-8621-cfee46ac293e service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:23:06.830301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1214/2423] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:23:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:23:10.555957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7d2618-40bf-45a0-a1cb-6f4d489db2b2 service nova] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:23:10.622659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7d2618-40bf-45a0-a1cb-6f4d489db2b2 service nova] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:23:10.628252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:23:10.628927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:23:10.629217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:23:10.629505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:23:10.631290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:23:10.654953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:23:10.656590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:23:10.667194 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1210/2424] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:23:10 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:23:14.190706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3294f4-de86-47a0-8621-cfee46ac293e service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:23:14.202227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3294f4-de86-47a0-8621-cfee46ac293e service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:23:14.207526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3397d663-b891-4c7b-b787-ad71b60fe1cc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:23:14.208365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3397d663-b891-4c7b-b787-ad71b60fe1cc service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:23:14.208716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3397d663-b891-4c7b-b787-ad71b60fe1cc service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:23:14.209425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3397d663-b891-4c7b-b787-ad71b60fe1cc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:23:14.211359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3397d663-b891-4c7b-b787-ad71b60fe1cc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:23:14.224440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3397d663-b891-4c7b-b787-ad71b60fe1cc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:23:14.226328 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3397d663-b891-4c7b-b787-ad71b60fe1cc service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:23:14.238175 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1215/2425] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:23:14 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:23:14.313594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:23:14.315245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:23:14.337786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a8e9a2-a950-43fb-882c-91c26c56c962 service placement] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:23:14.342152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:23:14.343035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:23:14.343323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:23:14.343602 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] PATH_INFO: `/v3/users/0a1368d42efc4543a908a61c48ec4472` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:23:14.358766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0a1368d42efc4543a908a61c48ec4472)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:23:14.361313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:23:14.524022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0a1368d42efc4543a908a61c48ec4472'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:14.524681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0a1368d42efc4543a908a61c48ec4472'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:14.531458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0a1368d42efc4543a908a61c48ec4472'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:14.584785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0a1368d42efc4543a908a61c48ec4472'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:14.585414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0a1368d42efc4543a908a61c48ec4472'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:14.603754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1211/2426] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:23:14 2022] DELETE /identity/v3/users/0a1368d42efc4543a908a61c48ec4472 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:23:14.843723 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3397d663-b891-4c7b-b787-ad71b60fe1cc service placement] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:23:14.855439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3397d663-b891-4c7b-b787-ad71b60fe1cc service placement] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:23:14.911958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:23:14.912855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:23:14.913142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:23:14.913424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] PATH_INFO: `/v3/projects/75273f09637240f3a5cc7129927765f1` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:23:14.914602 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75273f09637240f3a5cc7129927765f1)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:23:14.916531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:23:15.053249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75273f09637240f3a5cc7129927765f1', 'request_id': 'req-b2e7d72b-2588-49d9-ba50-de68acc24f44'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.056785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:23:15.138966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75273f09637240f3a5cc7129927765f1', 'request_id': 'req-b2e7d72b-2588-49d9-ba50-de68acc24f44'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.147157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75273f09637240f3a5cc7129927765f1', 'request_id': 'req-b2e7d72b-2588-49d9-ba50-de68acc24f44'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.159853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] The token cache is being invalidate because project 75273f09637240f3a5cc7129927765f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:23:15.160264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.160951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.162493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1216/2427] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:23:14 2022] DELETE /identity/v3/projects/75273f09637240f3a5cc7129927765f1 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:23:15.166461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:23:15.167886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:23:15.181404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a5fba0-e68d-41cc-88c7-489c43006e21 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:23:15.219130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:23:15.238448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:23:15.242393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:23:15.242393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:23:15.242393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] PATH_INFO: `/v3/users/6a2f463538ae433ea7a71b95721e8378` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:23:15.252570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a2f463538ae433ea7a71b95721e8378)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:23:15.254587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:23:15.439747 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a2f463538ae433ea7a71b95721e8378'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.440383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a2f463538ae433ea7a71b95721e8378'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.447141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a2f463538ae433ea7a71b95721e8378'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.504585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a2f463538ae433ea7a71b95721e8378'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.505325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a2f463538ae433ea7a71b95721e8378'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.522495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1212/2428] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:23:15 2022] DELETE /identity/v3/users/6a2f463538ae433ea7a71b95721e8378 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:23:15.746219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:23:15.750313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:23:15.761137 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e7d72b-2588-49d9-ba50-de68acc24f44 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:23:15.818864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:23:15.819722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:23:15.820007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:23:15.820286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] PATH_INFO: `/v3/projects/eefd4f68b49a4490b468a288fec0d57e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:23:15.830354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] RBAC: Authorizing `identity:delete_project(project_id=eefd4f68b49a4490b468a288fec0d57e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:23:15.832299 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:23:15.927829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eefd4f68b49a4490b468a288fec0d57e', 'request_id': 'req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.984716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eefd4f68b49a4490b468a288fec0d57e', 'request_id': 'req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:15.992151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eefd4f68b49a4490b468a288fec0d57e', 'request_id': 'req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:16.002496 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] The token cache is being invalidate because project eefd4f68b49a4490b468a288fec0d57e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:23:16.002785 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:16.003459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:23:16.005044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1217/2429] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:23:15 2022] DELETE /identity/v3/projects/eefd4f68b49a4490b468a288fec0d57e => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:23:25.518960 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=40,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:23:25.999371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=61,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:24:17.173748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:24:17.175172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:24:17.232115 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3fa971-1506-49d6-8922-efeeff4a4b72 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:24:17.279111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c62f128-dd68-467f-9013-8eff41512fb5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:24:17.279755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c62f128-dd68-467f-9013-8eff41512fb5 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:24:17.280038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c62f128-dd68-467f-9013-8eff41512fb5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:24:17.280313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c62f128-dd68-467f-9013-8eff41512fb5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:24:17.282139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7c62f128-dd68-467f-9013-8eff41512fb5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application [None req-7c62f128-dd68-467f-9013-8eff41512fb5 service cinder] Could not find project: 75273f09637240f3a5cc7129927765f1.: keystone.exception.ProjectNotFound: Could not find project: 75273f09637240f3a5cc7129927765f1. Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Traceback (most recent call last): Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/auth.py", line 285, in get Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') Jun 10 13:24:17.304181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 145, in validate_token Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application token = self._validate_token(token_id) Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 13:24:17.305451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/token/provider.py", line 179, in _validate_token Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application token.mint(token_id, issued_at) Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 579, in mint Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self._validate_token_resources() Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 464, in _validate_token_resources Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application if self.project and not self.project.get('enabled'): Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/models/token_model.py", line 162, in project Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application self.__project = PROVIDERS.resource_api.get_project( Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jun 10 13:24:17.306561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.get_or_create( Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application with Lock( Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._enter() Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.creator() Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application created_value = creator( Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jun 10 13:24:17.307804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jun 10 13:24:17.309087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: 75273f09637240f3a5cc7129927765f1. Jun 10 13:24:17.309087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: ERROR keystone.server.flask.application Jun 10 13:24:17.309087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1213/2430] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:24:17 2022] GET /identity/v3/auth/tokens => generated 113 bytes in 132 msecs (HTTP/1.1 404) 5 headers in 184 bytes (1 switches on core 0) Jun 10 13:24:18.505416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:24:18.506822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:24:18.574469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6ddbd5-9d02-4a41-9f6b-ba55f400a90e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:24:18.579354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:24:18.579993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:24:18.580789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:24:18.581070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:24:18.582792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:24:18.593834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:24:18.595410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:24:18.605937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1218/2431] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:24:18 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:24:27.286153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=44 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:24:28.601891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=32 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:25:20.256121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c62f128-dd68-467f-9013-8eff41512fb5 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:25:20.257498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7c62f128-dd68-467f-9013-8eff41512fb5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:25:20.268374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c62f128-dd68-467f-9013-8eff41512fb5 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:25:20.272338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ba6ef5-41c3-4e9a-9a1e-f9b1e0b4128e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:25:20.273008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ba6ef5-41c3-4e9a-9a1e-f9b1e0b4128e service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:25:20.273291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ba6ef5-41c3-4e9a-9a1e-f9b1e0b4128e service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:25:20.273563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ba6ef5-41c3-4e9a-9a1e-f9b1e0b4128e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:25:20.275314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-76ba6ef5-41c3-4e9a-9a1e-f9b1e0b4128e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:25:20.333818 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ba6ef5-41c3-4e9a-9a1e-f9b1e0b4128e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:25:20.335258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ba6ef5-41c3-4e9a-9a1e-f9b1e0b4128e service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:25:20.344093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1214/2432] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:25:20 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:25:20.531751 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:25:20.533778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:25:20.602407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-500efee2-3c77-43bd-8264-8b31f2b5e68a service placement] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:25:20.606991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:25:20.611360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:25:20.611360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:25:20.611360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] PATH_INFO: `/v3/users/d9eb7d339d224a34908d0fc55e3be73c` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:25:20.611360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=d9eb7d339d224a34908d0fc55e3be73c)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:25:20.612293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:25:20.658930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ba6ef5-41c3-4e9a-9a1e-f9b1e0b4128e service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:25:20.666713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ba6ef5-41c3-4e9a-9a1e-f9b1e0b4128e service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:25:20.672421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c275d0a-b678-4609-aed1-dc25444b54f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:25:20.673661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c275d0a-b678-4609-aed1-dc25444b54f8 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:25:20.673951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c275d0a-b678-4609-aed1-dc25444b54f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:25:20.674228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c275d0a-b678-4609-aed1-dc25444b54f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:25:20.676050 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3c275d0a-b678-4609-aed1-dc25444b54f8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:25:20.744827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c275d0a-b678-4609-aed1-dc25444b54f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:25:20.746315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c275d0a-b678-4609-aed1-dc25444b54f8 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:25:20.764034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1215/2433] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:25:20 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:25:20.805923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd9eb7d339d224a34908d0fc55e3be73c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:20.806576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd9eb7d339d224a34908d0fc55e3be73c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:20.807204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd9eb7d339d224a34908d0fc55e3be73c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:20.814126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd9eb7d339d224a34908d0fc55e3be73c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:20.872127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd9eb7d339d224a34908d0fc55e3be73c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:20.889945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1219/2434] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:25:20 2022] DELETE /identity/v3/users/d9eb7d339d224a34908d0fc55e3be73c => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:25:20.894878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96778fa-a031-4b8f-b69e-95c2bb054a87 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:25:20.895170 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96778fa-a031-4b8f-b69e-95c2bb054a87 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:25:20.895452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96778fa-a031-4b8f-b69e-95c2bb054a87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:25:20.921322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-d96778fa-a031-4b8f-b69e-95c2bb054a87 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:25:20.922580 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d96778fa-a031-4b8f-b69e-95c2bb054a87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:25:20.982337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1216/2435] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:25:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:25:20.993018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:25:20.998575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:25:21.055708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa85b55f-a054-4880-bec7-7cc7b4f4c7ab admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:25:21.059739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bf8e55-6d10-4215-b97b-de31cee00a16 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:25:21.060385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bf8e55-6d10-4215-b97b-de31cee00a16 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:25:21.060669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bf8e55-6d10-4215-b97b-de31cee00a16 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:25:21.060979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68bf8e55-6d10-4215-b97b-de31cee00a16 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:25:21.070449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bf8e55-6d10-4215-b97b-de31cee00a16 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:25:21.071910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68bf8e55-6d10-4215-b97b-de31cee00a16 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:25:21.082825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1220/2436] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:25:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:25:21.440399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96778fa-a031-4b8f-b69e-95c2bb054a87 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:25:21.441863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d96778fa-a031-4b8f-b69e-95c2bb054a87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:25:21.454910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96778fa-a031-4b8f-b69e-95c2bb054a87 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:25:21.459590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:25:21.460452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:25:21.460732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:25:21.469404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] PATH_INFO: `/v3/projects/534db947c7aa4d68ac59cdfd7843f6e8` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:25:21.469404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=534db947c7aa4d68ac59cdfd7843f6e8)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:25:21.469404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:25:21.474298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:25:21.580313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '534db947c7aa4d68ac59cdfd7843f6e8', 'request_id': 'req-12c243e0-161e-47e1-8567-fa4f7993d5e7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:21.590265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '534db947c7aa4d68ac59cdfd7843f6e8', 'request_id': 'req-12c243e0-161e-47e1-8567-fa4f7993d5e7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:21.597337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '534db947c7aa4d68ac59cdfd7843f6e8', 'request_id': 'req-12c243e0-161e-47e1-8567-fa4f7993d5e7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:21.664650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] The token cache is being invalidate because project 534db947c7aa4d68ac59cdfd7843f6e8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:25:21.664650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:21.664650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:25:21.665078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1217/2437] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:25:21 2022] DELETE /identity/v3/projects/534db947c7aa4d68ac59cdfd7843f6e8 => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:25:31.077533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:25:31.657147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=21,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:26:05.362611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bf8e55-6d10-4215-b97b-de31cee00a16 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:26:05.371650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68bf8e55-6d10-4215-b97b-de31cee00a16 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:26:05.419434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:26:05.420065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:26:05.420348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:26:05.420626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:26:05.423021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:26:05.490731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:26:05.492167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:26:05.502851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1221/2438] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:26:05 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:26:15.498066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=54 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:26:24.182979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:26:24.184863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:26:24.200466 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c243e0-161e-47e1-8567-fa4f7993d5e7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:26:24.204200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33666498-bb59-4814-ab19-257258ceefab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:26:24.204883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33666498-bb59-4814-ab19-257258ceefab service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:26:24.205318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33666498-bb59-4814-ab19-257258ceefab service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:26:24.205720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33666498-bb59-4814-ab19-257258ceefab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:26:24.208354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-33666498-bb59-4814-ab19-257258ceefab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:26:24.218623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33666498-bb59-4814-ab19-257258ceefab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:26:24.220627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33666498-bb59-4814-ab19-257258ceefab service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:26:24.228682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1218/2439] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:26:24 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:26:34.225057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=6 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:27:20.799708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:27:20.801178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:27:20.856861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541b6aec-2ba9-4d48-9692-acadbf4756c4 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:27:20.860432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee8faa5-214e-4bfc-8349-d54456754f20 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:27:20.861097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee8faa5-214e-4bfc-8349-d54456754f20 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:27:20.861381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee8faa5-214e-4bfc-8349-d54456754f20 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:27:20.861657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee8faa5-214e-4bfc-8349-d54456754f20 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:27:20.863416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-dee8faa5-214e-4bfc-8349-d54456754f20 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:27:20.928557 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee8faa5-214e-4bfc-8349-d54456754f20 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:27:20.930019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee8faa5-214e-4bfc-8349-d54456754f20 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:27:20.947708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1222/2440] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:27:20 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:27:20.991079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33666498-bb59-4814-ab19-257258ceefab service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:27:21.000134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33666498-bb59-4814-ab19-257258ceefab service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:27:21.004173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a326f313-3534-4f17-b957-4b0df2712446 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:27:21.004835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326f313-3534-4f17-b957-4b0df2712446 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:27:21.005123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326f313-3534-4f17-b957-4b0df2712446 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:27:21.005400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a326f313-3534-4f17-b957-4b0df2712446 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:27:21.007110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a326f313-3534-4f17-b957-4b0df2712446 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:27:21.085334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a326f313-3534-4f17-b957-4b0df2712446 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:27:21.086701 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a326f313-3534-4f17-b957-4b0df2712446 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:27:21.104638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1219/2441] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:27:20 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:27:21.147012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee8faa5-214e-4bfc-8349-d54456754f20 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:27:21.155476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee8faa5-214e-4bfc-8349-d54456754f20 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:27:21.158980 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:27:21.159616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:27:21.159897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:27:21.160174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:27:21.161935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:27:21.228416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:27:21.229949 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:27:21.248090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1223/2442] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:27:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:27:31.102026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=41 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:27:31.245523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:28:20.338068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf4427e-e715-4079-956c-f587bae32763 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:28:20.339147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf4427e-e715-4079-956c-f587bae32763 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:28:20.339147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf4427e-e715-4079-956c-f587bae32763 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:28:20.365517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-1bf4427e-e715-4079-956c-f587bae32763 None None] MFA Rules not processed for user `b1d1712ea523498f92f258383863b906`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:28:20.366723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1bf4427e-e715-4079-956c-f587bae32763 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:28:20.419244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1220/2443] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:28:20 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:28:20.711492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:28:20.712981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:28:20.733905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89765249-a553-486d-a0cf-ba661bd3b233 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:28:20.737984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:28:20.738620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:28:20.738906 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:28:20.739186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:28:20.741191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:28:20.751471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:28:20.752943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:28:20.760696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1224/2444] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:28:20 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:28:22.204872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf4427e-e715-4079-956c-f587bae32763 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:28:22.206249 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1bf4427e-e715-4079-956c-f587bae32763 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:28:22.276560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf4427e-e715-4079-956c-f587bae32763 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:28:22.281437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c883526-927e-4f4a-b35b-33d5a3c69be7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:28:22.282132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c883526-927e-4f4a-b35b-33d5a3c69be7 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:28:22.282445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c883526-927e-4f4a-b35b-33d5a3c69be7 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:28:22.282726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c883526-927e-4f4a-b35b-33d5a3c69be7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:28:22.284482 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8c883526-927e-4f4a-b35b-33d5a3c69be7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:28:22.296054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c883526-927e-4f4a-b35b-33d5a3c69be7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:28:22.298245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c883526-927e-4f4a-b35b-33d5a3c69be7 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:28:22.308375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1221/2445] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:28:22 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:28:29.020620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:28:29.023022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:28:29.067840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134492db-2fcb-49d7-aee6-032ebe5f023e service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:28:29.073663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472d05ba-857b-4759-a8e6-d3d25128b304 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:28:29.074377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472d05ba-857b-4759-a8e6-d3d25128b304 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:28:29.074614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472d05ba-857b-4759-a8e6-d3d25128b304 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:28:29.074893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472d05ba-857b-4759-a8e6-d3d25128b304 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:28:29.076793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-472d05ba-857b-4759-a8e6-d3d25128b304 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:28:29.090376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472d05ba-857b-4759-a8e6-d3d25128b304 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:28:29.092037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472d05ba-857b-4759-a8e6-d3d25128b304 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:28:29.103714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1225/2446] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:28:29 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:28:32.304500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=60 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:28:35.365275 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c883526-927e-4f4a-b35b-33d5a3c69be7 service placement] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:28:35.378664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c883526-927e-4f4a-b35b-33d5a3c69be7 service placement] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:28:35.384132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a82a025-a2ce-41ba-a2c9-f32c07c48ce3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:28:35.384785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a82a025-a2ce-41ba-a2c9-f32c07c48ce3 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:28:35.385096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a82a025-a2ce-41ba-a2c9-f32c07c48ce3 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:28:35.385376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a82a025-a2ce-41ba-a2c9-f32c07c48ce3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:28:35.387130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3a82a025-a2ce-41ba-a2c9-f32c07c48ce3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:28:35.399854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a82a025-a2ce-41ba-a2c9-f32c07c48ce3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:28:35.401497 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a82a025-a2ce-41ba-a2c9-f32c07c48ce3 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:28:35.412010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1222/2447] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:28:35 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:28:39.098838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=23 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:28:39.399769 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472d05ba-857b-4759-a8e6-d3d25128b304 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:28:39.411252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472d05ba-857b-4759-a8e6-d3d25128b304 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:28:39.416683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2090ade-17b0-4d31-94db-13e2754e53d3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:28:39.417353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2090ade-17b0-4d31-94db-13e2754e53d3 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:28:39.417638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2090ade-17b0-4d31-94db-13e2754e53d3 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:28:39.417920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2090ade-17b0-4d31-94db-13e2754e53d3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:28:39.419674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e2090ade-17b0-4d31-94db-13e2754e53d3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:28:39.431489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2090ade-17b0-4d31-94db-13e2754e53d3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:28:39.433100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2090ade-17b0-4d31-94db-13e2754e53d3 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:28:39.443651 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1226/2448] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:28:39 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:28:45.407840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=6 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:28:49.439341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:29:21.541921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a82a025-a2ce-41ba-a2c9-f32c07c48ce3 service nova] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:29:21.595536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a82a025-a2ce-41ba-a2c9-f32c07c48ce3 service nova] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:29:21.600878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eebf0b-20b4-43f5-98c5-b4b8893a783e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:29:21.601524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eebf0b-20b4-43f5-98c5-b4b8893a783e service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:29:21.601810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eebf0b-20b4-43f5-98c5-b4b8893a783e service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:29:21.602089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eebf0b-20b4-43f5-98c5-b4b8893a783e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:29:21.603902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-31eebf0b-20b4-43f5-98c5-b4b8893a783e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:29:21.625218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31eebf0b-20b4-43f5-98c5-b4b8893a783e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:29:21.627421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31eebf0b-20b4-43f5-98c5-b4b8893a783e service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:29:21.637808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1223/2449] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:29:21 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:29:31.633594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=28 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:31:05.739489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2090ade-17b0-4d31-94db-13e2754e53d3 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:31:05.767772 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2090ade-17b0-4d31-94db-13e2754e53d3 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:31:05.771362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa46325-9495-4562-9f94-8c3c755f0481 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:31:05.771995 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa46325-9495-4562-9f94-8c3c755f0481 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:31:05.772280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa46325-9495-4562-9f94-8c3c755f0481 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:31:05.772559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa46325-9495-4562-9f94-8c3c755f0481 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:31:05.783200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa46325-9495-4562-9f94-8c3c755f0481 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:31:05.784631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa46325-9495-4562-9f94-8c3c755f0481 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:31:05.796224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1227/2450] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:31:05 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:31:15.790691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=15 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:31:22.238332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eebf0b-20b4-43f5-98c5-b4b8893a783e service placement] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:31:22.258714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eebf0b-20b4-43f5-98c5-b4b8893a783e service placement] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:31:22.262622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b740c4f-f46c-4687-bd81-140be6a44df0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:31:22.263505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b740c4f-f46c-4687-bd81-140be6a44df0 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:31:22.263790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b740c4f-f46c-4687-bd81-140be6a44df0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:31:22.264084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b740c4f-f46c-4687-bd81-140be6a44df0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:31:22.266000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4b740c4f-f46c-4687-bd81-140be6a44df0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:31:22.294066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b740c4f-f46c-4687-bd81-140be6a44df0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:31:22.295515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b740c4f-f46c-4687-bd81-140be6a44df0 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:31:22.304734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1224/2451] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:31:22 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:31:26.166388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa46325-9495-4562-9f94-8c3c755f0481 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:31:26.175608 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa46325-9495-4562-9f94-8c3c755f0481 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:31:26.179247 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0da98b-5f66-4c03-baed-cbdaaf86c682 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:31:26.179879 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0da98b-5f66-4c03-baed-cbdaaf86c682 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:31:26.180163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0da98b-5f66-4c03-baed-cbdaaf86c682 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:31:26.180446 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0da98b-5f66-4c03-baed-cbdaaf86c682 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:31:26.187531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0da98b-5f66-4c03-baed-cbdaaf86c682 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:31:26.188994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c0da98b-5f66-4c03-baed-cbdaaf86c682 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:31:26.196844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1228/2452] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:31:26 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 31 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:31:32.300330 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=15 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:31:36.193246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:32:18.202587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b740c4f-f46c-4687-bd81-140be6a44df0 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:18.211979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b740c4f-f46c-4687-bd81-140be6a44df0 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:18.215473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:18.216104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:18.216387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:18.216661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:18.218460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:18.280052 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:18.281534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:18.290915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1225/2453] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:32:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:32:21.636021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0da98b-5f66-4c03-baed-cbdaaf86c682 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:21.652757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c0da98b-5f66-4c03-baed-cbdaaf86c682 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:21.660466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876432c1-4bca-4f71-9cd7-4ff022dc5402 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:21.661378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876432c1-4bca-4f71-9cd7-4ff022dc5402 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:21.661956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876432c1-4bca-4f71-9cd7-4ff022dc5402 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:21.662409 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876432c1-4bca-4f71-9cd7-4ff022dc5402 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:21.671160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876432c1-4bca-4f71-9cd7-4ff022dc5402 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:21.673135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876432c1-4bca-4f71-9cd7-4ff022dc5402 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:21.687618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1229/2454] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:32:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:32:22.009290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:22.010717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:22.025672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce55464-05ca-4120-b3d8-e32ba7e1dc8c service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:22.030748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41e8644-03b3-41d5-9d5d-dadb94e5e837 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:22.031392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41e8644-03b3-41d5-9d5d-dadb94e5e837 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:22.031683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41e8644-03b3-41d5-9d5d-dadb94e5e837 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:22.031965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41e8644-03b3-41d5-9d5d-dadb94e5e837 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:22.033761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f41e8644-03b3-41d5-9d5d-dadb94e5e837 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:22.046936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41e8644-03b3-41d5-9d5d-dadb94e5e837 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:22.048539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41e8644-03b3-41d5-9d5d-dadb94e5e837 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:22.059339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1226/2455] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:32:22 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:32:22.131942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876432c1-4bca-4f71-9cd7-4ff022dc5402 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:22.133393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-876432c1-4bca-4f71-9cd7-4ff022dc5402 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:22.147113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876432c1-4bca-4f71-9cd7-4ff022dc5402 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:22.151778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:22.152634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:22.152950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:22.153228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] PATH_INFO: `/v3/users/9c8b2c27b69a4776ab6b7b7820da02d9` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:22.167269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c8b2c27b69a4776ab6b7b7820da02d9)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:22.169262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:22.317666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c8b2c27b69a4776ab6b7b7820da02d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.318314 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c8b2c27b69a4776ab6b7b7820da02d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.318921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c8b2c27b69a4776ab6b7b7820da02d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.326387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c8b2c27b69a4776ab6b7b7820da02d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.390122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-95e33f76-7034-4cc4-a15d-ae7169437085 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c8b2c27b69a4776ab6b7b7820da02d9'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.408061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1230/2456] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:32:22 2022] DELETE /identity/v3/users/9c8b2c27b69a4776ab6b7b7820da02d9 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:32:22.646487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41e8644-03b3-41d5-9d5d-dadb94e5e837 service placement] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:22.660022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41e8644-03b3-41d5-9d5d-dadb94e5e837 service placement] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:22.727149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:22.729027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:22.729459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:22.729881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] PATH_INFO: `/v3/projects/6dace38cecd44f87becbf6dc213851b3` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:22.731664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6dace38cecd44f87becbf6dc213851b3)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:22.734775 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:22.839353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6dace38cecd44f87becbf6dc213851b3', 'request_id': 'req-6a3bc94a-ad4e-4b7c-91bd-888771356634'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.845488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6dace38cecd44f87becbf6dc213851b3', 'request_id': 'req-6a3bc94a-ad4e-4b7c-91bd-888771356634'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.851135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6dace38cecd44f87becbf6dc213851b3', 'request_id': 'req-6a3bc94a-ad4e-4b7c-91bd-888771356634'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.918824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] The token cache is being invalidate because project 6dace38cecd44f87becbf6dc213851b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:32:22.919050 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.919675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6a3bc94a-ad4e-4b7c-91bd-888771356634 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:32:22.921255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1227/2457] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:32:22 2022] DELETE /identity/v3/projects/6dace38cecd44f87becbf6dc213851b3 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:32:22.927078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32916510-3f88-4a7a-ad4b-8ca8db4b768f None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:22.927534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32916510-3f88-4a7a-ad4b-8ca8db4b768f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:22.927646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32916510-3f88-4a7a-ad4b-8ca8db4b768f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:22.974358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-32916510-3f88-4a7a-ad4b-8ca8db4b768f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:32:22.975562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-32916510-3f88-4a7a-ad4b-8ca8db4b768f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:23.032262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1231/2458] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:32:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:32:23.040042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fb5556-6673-4acf-a919-e11620df900d None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:23.040567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fb5556-6673-4acf-a919-e11620df900d None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:23.040567 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97fb5556-6673-4acf-a919-e11620df900d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:23.059444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-97fb5556-6673-4acf-a919-e11620df900d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:32:23.060660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-97fb5556-6673-4acf-a919-e11620df900d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:23.069066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1228/2459] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:32:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:32:23.073638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32916510-3f88-4a7a-ad4b-8ca8db4b768f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:23.086827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32916510-3f88-4a7a-ad4b-8ca8db4b768f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:23.092165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2543f2-1fbf-4760-ab23-7fe49cb4783e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:23.092896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2543f2-1fbf-4760-ab23-7fe49cb4783e admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:23.093182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2543f2-1fbf-4760-ab23-7fe49cb4783e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:23.093464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2543f2-1fbf-4760-ab23-7fe49cb4783e admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:23.094192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2543f2-1fbf-4760-ab23-7fe49cb4783e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:32:23.094501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2543f2-1fbf-4760-ab23-7fe49cb4783e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:23.096119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2543f2-1fbf-4760-ab23-7fe49cb4783e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:23.108808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1232/2460] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:32:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:32:23.113147 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fb5556-6673-4acf-a919-e11620df900d None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:23.118741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:32:23.128868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97fb5556-6673-4acf-a919-e11620df900d None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:23.130724 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fdc417-b274-4e90-895a-07f53a13aa41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:23.131386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fdc417-b274-4e90-895a-07f53a13aa41 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:23.131731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fdc417-b274-4e90-895a-07f53a13aa41 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:23.131980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fdc417-b274-4e90-895a-07f53a13aa41 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:23.132742 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fdc417-b274-4e90-895a-07f53a13aa41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:23.134691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fdc417-b274-4e90-895a-07f53a13aa41 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:23.214802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1229/2461] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:32:23 2022] POST /identity/v3/projects => generated 385 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:32:23.218881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2543f2-1fbf-4760-ab23-7fe49cb4783e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:23.229445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2543f2-1fbf-4760-ab23-7fe49cb4783e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:23.237105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:32:23.284294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1957638-1a2e-40df-8efe-78e97e9fa01f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:23.284984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1957638-1a2e-40df-8efe-78e97e9fa01f admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:23.285272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1957638-1a2e-40df-8efe-78e97e9fa01f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:23.285562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1957638-1a2e-40df-8efe-78e97e9fa01f admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:23.286249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1957638-1a2e-40df-8efe-78e97e9fa01f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:23.288165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1957638-1a2e-40df-8efe-78e97e9fa01f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:23.372895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1233/2462] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:32:23 2022] POST /identity/v3/users => generated 293 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:32:23.377073 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fdc417-b274-4e90-895a-07f53a13aa41 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:23.388423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fdc417-b274-4e90-895a-07f53a13aa41 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:23.393134 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ea41da-4aaf-4787-86bd-66b624e9672f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:23.393787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ea41da-4aaf-4787-86bd-66b624e9672f admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:23.394067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ea41da-4aaf-4787-86bd-66b624e9672f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:23.394344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ea41da-4aaf-4787-86bd-66b624e9672f admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:23.395065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ea41da-4aaf-4787-86bd-66b624e9672f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:23.396678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ea41da-4aaf-4787-86bd-66b624e9672f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:23.407266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1230/2463] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:32:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:32:23.411339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1957638-1a2e-40df-8efe-78e97e9fa01f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:23.421183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1957638-1a2e-40df-8efe-78e97e9fa01f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:23.426676 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-283b0294-5a36-4f76-a70d-c5916b6cd98f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:23.427361 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283b0294-5a36-4f76-a70d-c5916b6cd98f admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:23.427648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283b0294-5a36-4f76-a70d-c5916b6cd98f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:23.427930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-283b0294-5a36-4f76-a70d-c5916b6cd98f admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:23.428673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283b0294-5a36-4f76-a70d-c5916b6cd98f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:23.430364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-283b0294-5a36-4f76-a70d-c5916b6cd98f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:23.443638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1234/2464] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:32:23 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:32:23.448426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ea41da-4aaf-4787-86bd-66b624e9672f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:23.460055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ea41da-4aaf-4787-86bd-66b624e9672f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:23.465340 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3cc0f3-2d69-4f3b-bcf9-c214f2cda373 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:23.466161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3cc0f3-2d69-4f3b-bcf9-c214f2cda373 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:23.466441 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3cc0f3-2d69-4f3b-bcf9-c214f2cda373 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:23.466716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3cc0f3-2d69-4f3b-bcf9-c214f2cda373 admin admin] PATH_INFO: `/v3/projects/eee8e4f43f484abdb588a8f3966d74ad/users/434ae422b68e4145ad8ecac27f450cbc/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:23.481000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3cc0f3-2d69-4f3b-bcf9-c214f2cda373 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eee8e4f43f484abdb588a8f3966d74ad, user_id=434ae422b68e4145ad8ecac27f450cbc, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:23.483654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3cc0f3-2d69-4f3b-bcf9-c214f2cda373 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:23.569849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1231/2465] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:32:23 2022] PUT /identity/v3/projects/eee8e4f43f484abdb588a8f3966d74ad/users/434ae422b68e4145ad8ecac27f450cbc/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:32:23.575458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65999c-e254-41b5-91e6-cd0a7cacb6a0 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:23.575841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65999c-e254-41b5-91e6-cd0a7cacb6a0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:23.576131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b65999c-e254-41b5-91e6-cd0a7cacb6a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:23.597105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-8b65999c-e254-41b5-91e6-cd0a7cacb6a0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:32:23.598580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8b65999c-e254-41b5-91e6-cd0a7cacb6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:23.656042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1235/2466] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:32:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:32:23.665423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3cc0f3-2d69-4f3b-bcf9-c214f2cda373 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:23.667122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1a3cc0f3-2d69-4f3b-bcf9-c214f2cda373 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:23.737098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3cc0f3-2d69-4f3b-bcf9-c214f2cda373 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:23.741129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb37c230-43c5-465e-b401-5cee60978f42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:23.741770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb37c230-43c5-465e-b401-5cee60978f42 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:23.742051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb37c230-43c5-465e-b401-5cee60978f42 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:23.742327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb37c230-43c5-465e-b401-5cee60978f42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:23.753809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb37c230-43c5-465e-b401-5cee60978f42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:23.755260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb37c230-43c5-465e-b401-5cee60978f42 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:23.765673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1232/2467] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:32:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:32:29.005134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b55975-c0b5-4004-8f76-6385ef7d4403 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:29.005695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b55975-c0b5-4004-8f76-6385ef7d4403 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:29.005695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42b55975-c0b5-4004-8f76-6385ef7d4403 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:29.035089 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-42b55975-c0b5-4004-8f76-6385ef7d4403 None None] MFA Rules not processed for user `434ae422b68e4145ad8ecac27f450cbc`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:32:29.036654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-42b55975-c0b5-4004-8f76-6385ef7d4403 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:29.110567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1236/2468] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:32:29 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:32:29.124802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb37c230-43c5-465e-b401-5cee60978f42 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:32:29.126296 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-bb37c230-43c5-465e-b401-5cee60978f42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:29.229475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb37c230-43c5-465e-b401-5cee60978f42 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:32:29.236181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96cb545-6b90-4d18-9328-f8c79f7340bb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:32:29.236967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96cb545-6b90-4d18-9328-f8c79f7340bb service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:29.237255 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96cb545-6b90-4d18-9328-f8c79f7340bb service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:29.237533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96cb545-6b90-4d18-9328-f8c79f7340bb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:29.247437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96cb545-6b90-4d18-9328-f8c79f7340bb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:32:29.248968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96cb545-6b90-4d18-9328-f8c79f7340bb service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:32:29.259074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1233/2469] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:32:29 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:32:31.552362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82b537-4539-4ff3-8f60-6cc4076fee80 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:32:31.552895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82b537-4539-4ff3-8f60-6cc4076fee80 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:32:31.553006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb82b537-4539-4ff3-8f60-6cc4076fee80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:32:31.574465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-fb82b537-4539-4ff3-8f60-6cc4076fee80 None None] MFA Rules not processed for user `434ae422b68e4145ad8ecac27f450cbc`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:32:31.575747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fb82b537-4539-4ff3-8f60-6cc4076fee80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:32:31.583097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1237/2470] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:32:31 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:32:39.254156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=96,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:32:41.566263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=91,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:33:22.518165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96cb545-6b90-4d18-9328-f8c79f7340bb service nova] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:33:22.518165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b96cb545-6b90-4d18-9328-f8c79f7340bb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:33:22.591930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96cb545-6b90-4d18-9328-f8c79f7340bb service nova] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:33:22.591930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e301964d-5e36-44b7-8fe8-f275d4602a49 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:33:22.591930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e301964d-5e36-44b7-8fe8-f275d4602a49 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:33:22.591930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e301964d-5e36-44b7-8fe8-f275d4602a49 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:33:22.592493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e301964d-5e36-44b7-8fe8-f275d4602a49 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:33:22.593460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e301964d-5e36-44b7-8fe8-f275d4602a49 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:33:22.657124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e301964d-5e36-44b7-8fe8-f275d4602a49 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:33:22.658612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e301964d-5e36-44b7-8fe8-f275d4602a49 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:33:22.668934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1234/2471] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:33:22 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:33:32.663382 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=53 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:34:29.177683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb82b537-4539-4ff3-8f60-6cc4076fee80 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:34:29.179081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-fb82b537-4539-4ff3-8f60-6cc4076fee80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:34:29.259504 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb82b537-4539-4ff3-8f60-6cc4076fee80 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:34:29.264487 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ec3486-c1df-4fdf-bad9-d7f4e9ab5131 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:34:29.265163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ec3486-c1df-4fdf-bad9-d7f4e9ab5131 service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:34:29.265453 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ec3486-c1df-4fdf-bad9-d7f4e9ab5131 service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:34:29.265737 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ec3486-c1df-4fdf-bad9-d7f4e9ab5131 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:34:29.267479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-03ec3486-c1df-4fdf-bad9-d7f4e9ab5131 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:34:29.278553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ec3486-c1df-4fdf-bad9-d7f4e9ab5131 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:34:29.280130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03ec3486-c1df-4fdf-bad9-d7f4e9ab5131 service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:34:29.290307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1238/2472] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:34:29 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:34:39.285696 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=37 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:36:18.366098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e301964d-5e36-44b7-8fe8-f275d4602a49 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:36:18.375384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e301964d-5e36-44b7-8fe8-f275d4602a49 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:36:18.379601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:36:18.380234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:36:18.380512 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:36:18.380817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:36:18.382674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:36:18.396486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:36:18.397993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:36:18.409373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1235/2473] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:36:18 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:36:20.062416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ec3486-c1df-4fdf-bad9-d7f4e9ab5131 service placement] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:36:20.077506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ec3486-c1df-4fdf-bad9-d7f4e9ab5131 service placement] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:36:20.084369 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beac9a2-7c52-44a3-b296-bfb9b824b86e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:36:20.085207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beac9a2-7c52-44a3-b296-bfb9b824b86e service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:36:20.086250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beac9a2-7c52-44a3-b296-bfb9b824b86e service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:36:20.086547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7beac9a2-7c52-44a3-b296-bfb9b824b86e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:36:20.088392 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7beac9a2-7c52-44a3-b296-bfb9b824b86e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:36:20.112626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7beac9a2-7c52-44a3-b296-bfb9b824b86e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:36:20.114730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7beac9a2-7c52-44a3-b296-bfb9b824b86e service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:36:20.127592 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1239/2474] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:36:20 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 66 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:36:22.162136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:36:22.163858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:36:22.176393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459bf5b6-cbc5-486c-8feb-2741b724b041 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:36:22.181026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376df241-d531-4f44-954b-4f60f73029c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:36:22.181945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376df241-d531-4f44-954b-4f60f73029c5 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:36:22.182327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376df241-d531-4f44-954b-4f60f73029c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:36:22.182690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376df241-d531-4f44-954b-4f60f73029c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:36:22.184797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-376df241-d531-4f44-954b-4f60f73029c5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:36:22.255057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376df241-d531-4f44-954b-4f60f73029c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:36:22.256456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376df241-d531-4f44-954b-4f60f73029c5 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:36:22.274239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1236/2475] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:36:22 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:36:28.148928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beac9a2-7c52-44a3-b296-bfb9b824b86e service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:36:28.158158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7beac9a2-7c52-44a3-b296-bfb9b824b86e service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:36:28.161820 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5575d6-289d-4539-8ea6-2561658fa592 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:36:28.162456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5575d6-289d-4539-8ea6-2561658fa592 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:36:28.162746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5575d6-289d-4539-8ea6-2561658fa592 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:36:28.163029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5575d6-289d-4539-8ea6-2561658fa592 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:36:28.164807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3b5575d6-289d-4539-8ea6-2561658fa592 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:36:28.174654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5575d6-289d-4539-8ea6-2561658fa592 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:36:28.176071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5575d6-289d-4539-8ea6-2561658fa592 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:36:28.184042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1240/2476] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:36:28 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:36:32.271671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=42 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:36:38.180571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=12 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:21.617809 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376df241-d531-4f44-954b-4f60f73029c5 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:21.627498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376df241-d531-4f44-954b-4f60f73029c5 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:21.631068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35f2a81-fa61-44da-a432-05606a5a8eae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:21.631717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35f2a81-fa61-44da-a432-05606a5a8eae service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:21.631999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35f2a81-fa61-44da-a432-05606a5a8eae service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:21.632271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35f2a81-fa61-44da-a432-05606a5a8eae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:21.633989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d35f2a81-fa61-44da-a432-05606a5a8eae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:21.711604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35f2a81-fa61-44da-a432-05606a5a8eae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:21.713071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35f2a81-fa61-44da-a432-05606a5a8eae service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:21.732241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1237/2477] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:37:28.331262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5575d6-289d-4539-8ea6-2561658fa592 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:28.339970 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5575d6-289d-4539-8ea6-2561658fa592 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:28.344441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43413eb4-bf88-44fa-a740-31143540d720 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:28.345112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43413eb4-bf88-44fa-a740-31143540d720 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:28.349138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43413eb4-bf88-44fa-a740-31143540d720 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:28.349138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43413eb4-bf88-44fa-a740-31143540d720 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:28.349138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-43413eb4-bf88-44fa-a740-31143540d720 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:28.383993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43413eb4-bf88-44fa-a740-31143540d720 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:28.385478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43413eb4-bf88-44fa-a740-31143540d720 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:28.404650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1241/2478] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:37:30.258694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35f2a81-fa61-44da-a432-05606a5a8eae service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:30.270042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35f2a81-fa61-44da-a432-05606a5a8eae service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:30.273963 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08963f17-7d60-4799-9151-ca2eb4490246 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:30.274624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08963f17-7d60-4799-9151-ca2eb4490246 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:30.274914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08963f17-7d60-4799-9151-ca2eb4490246 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:30.275197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08963f17-7d60-4799-9151-ca2eb4490246 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:30.288500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08963f17-7d60-4799-9151-ca2eb4490246 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:30.291782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08963f17-7d60-4799-9151-ca2eb4490246 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:30.302592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1238/2479] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:30 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:37:32.196068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43413eb4-bf88-44fa-a740-31143540d720 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:32.207890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43413eb4-bf88-44fa-a740-31143540d720 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:32.212979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c138692-db6c-4528-9240-e5e08bb3e32a service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:32.213626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c138692-db6c-4528-9240-e5e08bb3e32a service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:32.217846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c138692-db6c-4528-9240-e5e08bb3e32a service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:32.217846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c138692-db6c-4528-9240-e5e08bb3e32a service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:32.217846 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3c138692-db6c-4528-9240-e5e08bb3e32a service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:32.228072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c138692-db6c-4528-9240-e5e08bb3e32a service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:32.229781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c138692-db6c-4528-9240-e5e08bb3e32a service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:32.240710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1242/2480] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:32 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:37:32.309135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08963f17-7d60-4799-9151-ca2eb4490246 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:32.310586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-08963f17-7d60-4799-9151-ca2eb4490246 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:32.325301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08963f17-7d60-4799-9151-ca2eb4490246 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:32.330446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:32.331316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:32.331598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:32.331875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] PATH_INFO: `/v3/users/25f73d0bd4e64057a922d11be46eb04c` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:32.345785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] RBAC: Authorizing `identity:delete_user(user_id=25f73d0bd4e64057a922d11be46eb04c)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:32.347748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:32.525877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '25f73d0bd4e64057a922d11be46eb04c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:32.526545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '25f73d0bd4e64057a922d11be46eb04c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:32.533266 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '25f73d0bd4e64057a922d11be46eb04c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:32.632180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '25f73d0bd4e64057a922d11be46eb04c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:32.632992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-7f585e03-c0c0-47db-9871-3f11dfdc3621 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '25f73d0bd4e64057a922d11be46eb04c'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:32.650746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1239/2481] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:37:32 2022] DELETE /identity/v3/users/25f73d0bd4e64057a922d11be46eb04c => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:32.896793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c138692-db6c-4528-9240-e5e08bb3e32a service placement] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:32.911330 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c138692-db6c-4528-9240-e5e08bb3e32a service placement] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:32.972075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:32.972866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:32.973150 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:32.973427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] PATH_INFO: `/v3/projects/03423e3482614374afa8c929bd931262` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:32.983484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] RBAC: Authorizing `identity:delete_project(project_id=03423e3482614374afa8c929bd931262)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:32.985439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:33.101154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03423e3482614374afa8c929bd931262', 'request_id': 'req-9b11a446-8726-46d0-842a-8533e67abaea'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:33.192980 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03423e3482614374afa8c929bd931262', 'request_id': 'req-9b11a446-8726-46d0-842a-8533e67abaea'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:33.200568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03423e3482614374afa8c929bd931262', 'request_id': 'req-9b11a446-8726-46d0-842a-8533e67abaea'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:33.211818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] The token cache is being invalidate because project 03423e3482614374afa8c929bd931262 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:37:33.212104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:33.212808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-9b11a446-8726-46d0-842a-8533e67abaea admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:33.219280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1243/2482] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:37:32 2022] DELETE /identity/v3/projects/03423e3482614374afa8c929bd931262 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:33.220501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37b6b82-3139-4986-85b8-46729ed0da3a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:33.220816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37b6b82-3139-4986-85b8-46729ed0da3a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:33.221099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37b6b82-3139-4986-85b8-46729ed0da3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:33.247249 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a37b6b82-3139-4986-85b8-46729ed0da3a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:33.248375 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a37b6b82-3139-4986-85b8-46729ed0da3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:33.254095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:33.308229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1240/2483] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:33.316555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d277880a-0479-4b41-9437-4445d5ff1947 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:33.316876 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d277880a-0479-4b41-9437-4445d5ff1947 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:33.317162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d277880a-0479-4b41-9437-4445d5ff1947 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:33.336944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-d277880a-0479-4b41-9437-4445d5ff1947 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:33.337712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d277880a-0479-4b41-9437-4445d5ff1947 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:33.345714 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1244/2484] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:33.349949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37b6b82-3139-4986-85b8-46729ed0da3a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:33.359988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37b6b82-3139-4986-85b8-46729ed0da3a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:33.364676 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360fa6e1-dec9-4475-9ddd-6727bac0ef0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:33.365371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360fa6e1-dec9-4475-9ddd-6727bac0ef0a admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:33.365655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360fa6e1-dec9-4475-9ddd-6727bac0ef0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:33.365934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360fa6e1-dec9-4475-9ddd-6727bac0ef0a admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:33.366683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360fa6e1-dec9-4475-9ddd-6727bac0ef0a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:37:33.367017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360fa6e1-dec9-4475-9ddd-6727bac0ef0a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:33.368587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360fa6e1-dec9-4475-9ddd-6727bac0ef0a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:33.380993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1241/2485] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:37:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:37:33.385111 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d277880a-0479-4b41-9437-4445d5ff1947 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:33.394408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d277880a-0479-4b41-9437-4445d5ff1947 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:33.399134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ebe25a-cc0e-44cc-bee8-f0496013b480 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:33.399778 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ebe25a-cc0e-44cc-bee8-f0496013b480 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:33.400066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ebe25a-cc0e-44cc-bee8-f0496013b480 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:33.400345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ebe25a-cc0e-44cc-bee8-f0496013b480 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:33.401067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ebe25a-cc0e-44cc-bee8-f0496013b480 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:33.402983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40ebe25a-cc0e-44cc-bee8-f0496013b480 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:33.485771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:33.495656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1245/2486] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:37:33 2022] POST /identity/v3/projects => generated 369 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:37:33.502315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360fa6e1-dec9-4475-9ddd-6727bac0ef0a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:33.514515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360fa6e1-dec9-4475-9ddd-6727bac0ef0a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:33.570732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c421fa-dd72-4def-b5cd-9dc23455a17b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:33.571388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c421fa-dd72-4def-b5cd-9dc23455a17b admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:33.571672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c421fa-dd72-4def-b5cd-9dc23455a17b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:33.571946 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c421fa-dd72-4def-b5cd-9dc23455a17b admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:33.572626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c421fa-dd72-4def-b5cd-9dc23455a17b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:33.574675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c421fa-dd72-4def-b5cd-9dc23455a17b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:33.670704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1242/2487] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:37:33 2022] POST /identity/v3/users => generated 285 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:37:33.675545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ebe25a-cc0e-44cc-bee8-f0496013b480 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:33.686212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ebe25a-cc0e-44cc-bee8-f0496013b480 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:33.691562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0d3612-3e90-4c0d-950b-ca240583e5b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:33.692266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0d3612-3e90-4c0d-950b-ca240583e5b4 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:33.692821 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0d3612-3e90-4c0d-950b-ca240583e5b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:33.693100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0d3612-3e90-4c0d-950b-ca240583e5b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:33.693937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0d3612-3e90-4c0d-950b-ca240583e5b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:33.695534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0d3612-3e90-4c0d-950b-ca240583e5b4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:33.706086 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1246/2488] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:33.709952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c421fa-dd72-4def-b5cd-9dc23455a17b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:33.719821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c421fa-dd72-4def-b5cd-9dc23455a17b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:33.724454 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c9af83-9fc7-46c9-be6c-57cd284b206c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:33.725130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c9af83-9fc7-46c9-be6c-57cd284b206c admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:33.725421 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c9af83-9fc7-46c9-be6c-57cd284b206c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:33.725706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c9af83-9fc7-46c9-be6c-57cd284b206c admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:33.726432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c9af83-9fc7-46c9-be6c-57cd284b206c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:33.728039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c9af83-9fc7-46c9-be6c-57cd284b206c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:33.737702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1243/2489] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:33 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:33.741615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0d3612-3e90-4c0d-950b-ca240583e5b4 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:33.751825 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0d3612-3e90-4c0d-950b-ca240583e5b4 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:33.756424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f3cbf-7eb3-4d82-994d-96a235ed82c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:33.757322 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f3cbf-7eb3-4d82-994d-96a235ed82c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:33.757609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f3cbf-7eb3-4d82-994d-96a235ed82c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:33.757889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23f3cbf-7eb3-4d82-994d-96a235ed82c2 admin admin] PATH_INFO: `/v3/projects/019e03089b0549c4bdd21d7617633193/users/2e801c4301304030b65080d83f58fc5f/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:33.772868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f3cbf-7eb3-4d82-994d-96a235ed82c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=019e03089b0549c4bdd21d7617633193, user_id=2e801c4301304030b65080d83f58fc5f, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:33.775541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23f3cbf-7eb3-4d82-994d-96a235ed82c2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:33.832802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1247/2490] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:37:33 2022] PUT /identity/v3/projects/019e03089b0549c4bdd21d7617633193/users/2e801c4301304030b65080d83f58fc5f/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:33.846270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd19f7a-b973-480e-bd3a-ecd578e10f68 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:33.846563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd19f7a-b973-480e-bd3a-ecd578e10f68 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:33.846843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd19f7a-b973-480e-bd3a-ecd578e10f68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:33.872590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-bdd19f7a-b973-480e-bd3a-ecd578e10f68 None None] MFA Rules not processed for user `2e801c4301304030b65080d83f58fc5f`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:33.873835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-bdd19f7a-b973-480e-bd3a-ecd578e10f68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:33.936863 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1244/2491] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:33 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:33.941440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f3cbf-7eb3-4d82-994d-96a235ed82c2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:34.001886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23f3cbf-7eb3-4d82-994d-96a235ed82c2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:34.006587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20036ed-65a4-4172-8098-bf8c66602685 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:34.007176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20036ed-65a4-4172-8098-bf8c66602685 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:34.007466 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20036ed-65a4-4172-8098-bf8c66602685 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:34.007743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20036ed-65a4-4172-8098-bf8c66602685 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:34.008428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20036ed-65a4-4172-8098-bf8c66602685 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:34.010372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20036ed-65a4-4172-8098-bf8c66602685 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:34.109297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1248/2492] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:37:33 2022] POST /identity/v3/projects => generated 373 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:37:34.116915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd19f7a-b973-480e-bd3a-ecd578e10f68 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:34.127535 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd19f7a-b973-480e-bd3a-ecd578e10f68 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:34.131064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:34.184596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e0f494-549f-4889-9bfc-535dcb118b15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:34.185299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e0f494-549f-4889-9bfc-535dcb118b15 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:34.185819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e0f494-549f-4889-9bfc-535dcb118b15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:34.186128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e0f494-549f-4889-9bfc-535dcb118b15 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:34.187025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e0f494-549f-4889-9bfc-535dcb118b15 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:34.189483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e0f494-549f-4889-9bfc-535dcb118b15 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:34.248829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1245/2493] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:37:34 2022] POST /identity/v3/users => generated 293 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:37:34.252957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20036ed-65a4-4172-8098-bf8c66602685 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:34.263677 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20036ed-65a4-4172-8098-bf8c66602685 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:34.268701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3026c09d-5bf8-4fcd-931b-89ff9df98c79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:34.269393 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3026c09d-5bf8-4fcd-931b-89ff9df98c79 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:34.269679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3026c09d-5bf8-4fcd-931b-89ff9df98c79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:34.269958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3026c09d-5bf8-4fcd-931b-89ff9df98c79 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:34.270674 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3026c09d-5bf8-4fcd-931b-89ff9df98c79 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:34.272305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3026c09d-5bf8-4fcd-931b-89ff9df98c79 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:34.283387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1249/2494] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:34.287253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e0f494-549f-4889-9bfc-535dcb118b15 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:34.297271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e0f494-549f-4889-9bfc-535dcb118b15 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:34.301962 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7796b14-a03b-4e03-85e3-b5b243152769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:34.302841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7796b14-a03b-4e03-85e3-b5b243152769 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:34.303126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7796b14-a03b-4e03-85e3-b5b243152769 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:34.303405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7796b14-a03b-4e03-85e3-b5b243152769 admin admin] PATH_INFO: `/v3/projects/15fab61867d2491abeb30328bb06dbca/users/6caca382d16e44fdb47dafe5b95bc9eb/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:34.318293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7796b14-a03b-4e03-85e3-b5b243152769 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15fab61867d2491abeb30328bb06dbca, user_id=6caca382d16e44fdb47dafe5b95bc9eb, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:34.321015 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7796b14-a03b-4e03-85e3-b5b243152769 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:34.369973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1246/2495] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:37:34 2022] PUT /identity/v3/projects/15fab61867d2491abeb30328bb06dbca/users/6caca382d16e44fdb47dafe5b95bc9eb/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:37:34.373873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3026c09d-5bf8-4fcd-931b-89ff9df98c79 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:34.383891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3026c09d-5bf8-4fcd-931b-89ff9df98c79 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:34.441899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6db1d92d-f0c5-4156-a921-8fc1f6e8f261 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:34.442828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db1d92d-f0c5-4156-a921-8fc1f6e8f261 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:34.443136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db1d92d-f0c5-4156-a921-8fc1f6e8f261 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:34.443424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6db1d92d-f0c5-4156-a921-8fc1f6e8f261 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:34.444148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db1d92d-f0c5-4156-a921-8fc1f6e8f261 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:34.445801 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6db1d92d-f0c5-4156-a921-8fc1f6e8f261 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:34.456022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1250/2496] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:34 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:34.460143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7796b14-a03b-4e03-85e3-b5b243152769 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:34.470373 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7796b14-a03b-4e03-85e3-b5b243152769 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:34.474973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbc2ba2-9e3e-4b70-834b-710cde5fd815 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:34.475842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc2ba2-9e3e-4b70-834b-710cde5fd815 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:34.476131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc2ba2-9e3e-4b70-834b-710cde5fd815 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:34.476407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc2ba2-9e3e-4b70-834b-710cde5fd815 admin admin] PATH_INFO: `/v3/projects/15fab61867d2491abeb30328bb06dbca/users/6caca382d16e44fdb47dafe5b95bc9eb/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:34.478434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbc2ba2-9e3e-4b70-834b-710cde5fd815 admin admin] RBAC: Authorizing `identity:create_grant(project_id=15fab61867d2491abeb30328bb06dbca, user_id=6caca382d16e44fdb47dafe5b95bc9eb, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:34.481112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbc2ba2-9e3e-4b70-834b-710cde5fd815 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:34.491336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1247/2497] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:37:34 2022] PUT /identity/v3/projects/15fab61867d2491abeb30328bb06dbca/users/6caca382d16e44fdb47dafe5b95bc9eb/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:34.503918 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bcf500-eb19-43a2-9c2d-cf48d3a9747a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:34.504216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bcf500-eb19-43a2-9c2d-cf48d3a9747a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:34.504503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bcf500-eb19-43a2-9c2d-cf48d3a9747a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:34.532006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a4bcf500-eb19-43a2-9c2d-cf48d3a9747a None None] MFA Rules not processed for user `6caca382d16e44fdb47dafe5b95bc9eb`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:34.533287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a4bcf500-eb19-43a2-9c2d-cf48d3a9747a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:34.553915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:34.604412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1251/2498] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:34 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:34.614977 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbc2ba2-9e3e-4b70-834b-710cde5fd815 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:34.616623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ccbc2ba2-9e3e-4b70-834b-710cde5fd815 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:34.673490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbc2ba2-9e3e-4b70-834b-710cde5fd815 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:34.677096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f69184f-ebeb-4b47-b36e-a70ae1434077 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:34.677729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f69184f-ebeb-4b47-b36e-a70ae1434077 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:34.678012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f69184f-ebeb-4b47-b36e-a70ae1434077 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:34.678290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f69184f-ebeb-4b47-b36e-a70ae1434077 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:34.687557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f69184f-ebeb-4b47-b36e-a70ae1434077 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:34.689025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f69184f-ebeb-4b47-b36e-a70ae1434077 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:34.698459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1248/2499] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:34 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:37:35.597133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bcf500-eb19-43a2-9c2d-cf48d3a9747a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:35.658842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bcf500-eb19-43a2-9c2d-cf48d3a9747a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:35.663378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:35.664242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:35.664558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:35.664868 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] PATH_INFO: `/v3/users/2e801c4301304030b65080d83f58fc5f` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:35.666115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2e801c4301304030b65080d83f58fc5f)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:35.668064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:35.874533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2e801c4301304030b65080d83f58fc5f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:35.875208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2e801c4301304030b65080d83f58fc5f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:35.876105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2e801c4301304030b65080d83f58fc5f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:35.883081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2e801c4301304030b65080d83f58fc5f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:35.934716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2e801c4301304030b65080d83f58fc5f'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:35.953642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1252/2500] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:37:35 2022] DELETE /identity/v3/users/2e801c4301304030b65080d83f58fc5f => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:35.958283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca8343-5918-4855-b54a-9794eb44d0f3 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:35.958795 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca8343-5918-4855-b54a-9794eb44d0f3 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:35.958920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca8343-5918-4855-b54a-9794eb44d0f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:35.975740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-62ca8343-5918-4855-b54a-9794eb44d0f3 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:35.976967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-62ca8343-5918-4855-b54a-9794eb44d0f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:36.034654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1249/2501] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:36.044210 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:36.045629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:36.087891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:36.102461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4e009e-0f04-48dc-892e-b97ded559f31 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:36.106070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f42224-cd06-4661-84f9-f50f165896cf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:36.106715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f42224-cd06-4661-84f9-f50f165896cf service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:36.107005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f42224-cd06-4661-84f9-f50f165896cf service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:36.107284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f42224-cd06-4661-84f9-f50f165896cf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:36.116787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f42224-cd06-4661-84f9-f50f165896cf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:36.118227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f42224-cd06-4661-84f9-f50f165896cf service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:36.128114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1253/2502] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:37:36.544201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca8343-5918-4855-b54a-9794eb44d0f3 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:36.546230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-62ca8343-5918-4855-b54a-9794eb44d0f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:36.553685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:36.563935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca8343-5918-4855-b54a-9794eb44d0f3 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:36.574521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:36.574521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:36.574521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:36.574521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] PATH_INFO: `/v3/projects/019e03089b0549c4bdd21d7617633193` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:36.575316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] RBAC: Authorizing `identity:delete_project(project_id=019e03089b0549c4bdd21d7617633193)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:36.575316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:36.673360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '019e03089b0549c4bdd21d7617633193', 'request_id': 'req-815a302d-9170-4482-a477-17f711eff398'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:36.679568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '019e03089b0549c4bdd21d7617633193', 'request_id': 'req-815a302d-9170-4482-a477-17f711eff398'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:36.685487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '019e03089b0549c4bdd21d7617633193', 'request_id': 'req-815a302d-9170-4482-a477-17f711eff398'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:36.749989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] The token cache is being invalidate because project 019e03089b0549c4bdd21d7617633193 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:37:36.750304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:36.750957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:36.752501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1250/2503] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:37:36 2022] DELETE /identity/v3/projects/019e03089b0549c4bdd21d7617633193 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:36.756575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f42224-cd06-4661-84f9-f50f165896cf service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:36.766077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f42224-cd06-4661-84f9-f50f165896cf service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:36.822176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:36.823037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:36.823319 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:36.823617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] PATH_INFO: `/v3/users/6caca382d16e44fdb47dafe5b95bc9eb` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:36.824812 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6caca382d16e44fdb47dafe5b95bc9eb)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:36.826799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:37.026996 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6caca382d16e44fdb47dafe5b95bc9eb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.027679 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6caca382d16e44fdb47dafe5b95bc9eb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.028284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6caca382d16e44fdb47dafe5b95bc9eb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.035765 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6caca382d16e44fdb47dafe5b95bc9eb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.099045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-16aeec5b-25bb-49c6-ab39-2207a45ed1e8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6caca382d16e44fdb47dafe5b95bc9eb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.116544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1254/2504] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:37:36 2022] DELETE /identity/v3/users/6caca382d16e44fdb47dafe5b95bc9eb => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:37.553459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:37.555485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:37.570130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815a302d-9170-4482-a477-17f711eff398 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:37.627936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:37.628822 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:37.629109 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:37.629387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] PATH_INFO: `/v3/projects/15fab61867d2491abeb30328bb06dbca` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:37.630614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=15fab61867d2491abeb30328bb06dbca)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:37.632522 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:37.733067 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '15fab61867d2491abeb30328bb06dbca', 'request_id': 'req-d7995d66-8548-4299-996c-6650ed418f22'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.739154 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '15fab61867d2491abeb30328bb06dbca', 'request_id': 'req-d7995d66-8548-4299-996c-6650ed418f22'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.744577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '15fab61867d2491abeb30328bb06dbca', 'request_id': 'req-d7995d66-8548-4299-996c-6650ed418f22'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.807817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] The token cache is being invalidate because project 15fab61867d2491abeb30328bb06dbca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:37:37.808037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.808658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d7995d66-8548-4299-996c-6650ed418f22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:37.810250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1251/2505] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:37:37 2022] DELETE /identity/v3/projects/15fab61867d2491abeb30328bb06dbca => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:37.815712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3892b56b-c201-4451-92c3-ad3a951d105f None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:37.816198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3892b56b-c201-4451-92c3-ad3a951d105f None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:37.816311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3892b56b-c201-4451-92c3-ad3a951d105f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:37.833662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3892b56b-c201-4451-92c3-ad3a951d105f None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:37.834867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3892b56b-c201-4451-92c3-ad3a951d105f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:37.887310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:37.896102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1255/2506] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:37.907595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec06d4a-76dc-4418-b3a7-e1109dd04e78 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:37.907890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec06d4a-76dc-4418-b3a7-e1109dd04e78 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:37.908173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec06d4a-76dc-4418-b3a7-e1109dd04e78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:37.927236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-eec06d4a-76dc-4418-b3a7-e1109dd04e78 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:37.928557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-eec06d4a-76dc-4418-b3a7-e1109dd04e78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:37.937122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1252/2507] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:37.941402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3892b56b-c201-4451-92c3-ad3a951d105f None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:37.951453 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3892b56b-c201-4451-92c3-ad3a951d105f None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:37.955993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5ca387-c452-4ce6-b87c-515e01554b98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:37.956639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5ca387-c452-4ce6-b87c-515e01554b98 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:37.956972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5ca387-c452-4ce6-b87c-515e01554b98 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:37.957251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5ca387-c452-4ce6-b87c-515e01554b98 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:37.957971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5ca387-c452-4ce6-b87c-515e01554b98 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:37:37.958278 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5ca387-c452-4ce6-b87c-515e01554b98 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:37.959883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5ca387-c452-4ce6-b87c-515e01554b98 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:37.971222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1256/2508] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:37:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:37:37.975430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec06d4a-76dc-4418-b3a7-e1109dd04e78 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:37.985597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec06d4a-76dc-4418-b3a7-e1109dd04e78 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:37.990122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c80c1e-5ba5-4a45-89ba-1c5594ed1a17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:37.990759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c80c1e-5ba5-4a45-89ba-1c5594ed1a17 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:37.991043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c80c1e-5ba5-4a45-89ba-1c5594ed1a17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:37.991319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c80c1e-5ba5-4a45-89ba-1c5594ed1a17 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:37.992046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c80c1e-5ba5-4a45-89ba-1c5594ed1a17 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:37.993976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13c80c1e-5ba5-4a45-89ba-1c5594ed1a17 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:37.997987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:38.062519 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1253/2509] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:37:37 2022] POST /identity/v3/projects => generated 363 bytes in 87 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:37:38.066726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5ca387-c452-4ce6-b87c-515e01554b98 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:38.076998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5ca387-c452-4ce6-b87c-515e01554b98 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:38.131808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec842aff-9e15-4e98-be1e-6735d6e145db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:38.132467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec842aff-9e15-4e98-be1e-6735d6e145db admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:38.132752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec842aff-9e15-4e98-be1e-6735d6e145db admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:38.133062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec842aff-9e15-4e98-be1e-6735d6e145db admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:38.133741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec842aff-9e15-4e98-be1e-6735d6e145db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:38.135665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec842aff-9e15-4e98-be1e-6735d6e145db admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:38.209294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1257/2510] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:37:38 2022] POST /identity/v3/users => generated 282 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:37:38.213269 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c80c1e-5ba5-4a45-89ba-1c5594ed1a17 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:38.223658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c80c1e-5ba5-4a45-89ba-1c5594ed1a17 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:38.228254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf5e631-7685-4deb-84d4-0ce78e1ecd72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:38.228949 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf5e631-7685-4deb-84d4-0ce78e1ecd72 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:38.229238 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf5e631-7685-4deb-84d4-0ce78e1ecd72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:38.229520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf5e631-7685-4deb-84d4-0ce78e1ecd72 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:38.230247 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf5e631-7685-4deb-84d4-0ce78e1ecd72 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:38.231848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf5e631-7685-4deb-84d4-0ce78e1ecd72 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:38.241571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1254/2511] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:38 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:38.245244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec842aff-9e15-4e98-be1e-6735d6e145db admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:38.255609 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec842aff-9e15-4e98-be1e-6735d6e145db admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:38.260251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5a776c-829a-4373-bf0e-7a6df501cc71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:38.260936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a776c-829a-4373-bf0e-7a6df501cc71 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:38.261225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a776c-829a-4373-bf0e-7a6df501cc71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:38.261570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a776c-829a-4373-bf0e-7a6df501cc71 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:38.262292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5a776c-829a-4373-bf0e-7a6df501cc71 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:38.263896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5a776c-829a-4373-bf0e-7a6df501cc71 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:38.274313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1258/2512] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:38 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:38.278098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf5e631-7685-4deb-84d4-0ce78e1ecd72 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:38.288387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf5e631-7685-4deb-84d4-0ce78e1ecd72 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:38.292880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b753df5-f8df-40b9-901d-dfc31b28b3d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:38.293738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b753df5-f8df-40b9-901d-dfc31b28b3d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:38.294018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b753df5-f8df-40b9-901d-dfc31b28b3d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:38.294291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b753df5-f8df-40b9-901d-dfc31b28b3d1 admin admin] PATH_INFO: `/v3/projects/e57542abc6bc4b5faf5a05f91f2d417e/users/c59af2d25ae94ad1adfe0b9c67a59e19/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:38.308422 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b753df5-f8df-40b9-901d-dfc31b28b3d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e57542abc6bc4b5faf5a05f91f2d417e, user_id=c59af2d25ae94ad1adfe0b9c67a59e19, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:38.311108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b753df5-f8df-40b9-901d-dfc31b28b3d1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:38.409410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1255/2513] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:37:38 2022] PUT /identity/v3/projects/e57542abc6bc4b5faf5a05f91f2d417e/users/c59af2d25ae94ad1adfe0b9c67a59e19/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:38.420619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192fb495-fd3d-494b-aeff-13e54ae91677 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:38.420943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192fb495-fd3d-494b-aeff-13e54ae91677 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:38.421228 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192fb495-fd3d-494b-aeff-13e54ae91677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:38.447919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-192fb495-fd3d-494b-aeff-13e54ae91677 None None] MFA Rules not processed for user `c59af2d25ae94ad1adfe0b9c67a59e19`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:38.449158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-192fb495-fd3d-494b-aeff-13e54ae91677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:38.512030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1259/2514] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:38 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:38.517218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b753df5-f8df-40b9-901d-dfc31b28b3d1 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:38.578891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b753df5-f8df-40b9-901d-dfc31b28b3d1 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:38.583538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36ea778-a3ec-4597-92e6-2690fd98c6ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:38.584179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36ea778-a3ec-4597-92e6-2690fd98c6ef admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:38.584462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36ea778-a3ec-4597-92e6-2690fd98c6ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:38.584736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36ea778-a3ec-4597-92e6-2690fd98c6ef admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:38.585465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36ea778-a3ec-4597-92e6-2690fd98c6ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:38.587426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36ea778-a3ec-4597-92e6-2690fd98c6ef admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:38.672915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1256/2515] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:37:38 2022] POST /identity/v3/projects => generated 363 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:37:38.676976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192fb495-fd3d-494b-aeff-13e54ae91677 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:38.687828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-192fb495-fd3d-494b-aeff-13e54ae91677 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:38.737126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:38.744495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07529dc-cb99-4c0c-9f99-3d6781101ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:38.745163 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07529dc-cb99-4c0c-9f99-3d6781101ef7 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:38.745452 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07529dc-cb99-4c0c-9f99-3d6781101ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:38.745756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07529dc-cb99-4c0c-9f99-3d6781101ef7 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:38.746429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07529dc-cb99-4c0c-9f99-3d6781101ef7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:38.751237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e07529dc-cb99-4c0c-9f99-3d6781101ef7 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:38.849952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1260/2516] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:37:38 2022] POST /identity/v3/users => generated 288 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:37:38.854066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36ea778-a3ec-4597-92e6-2690fd98c6ef admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:38.864909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36ea778-a3ec-4597-92e6-2690fd98c6ef admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:38.869556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bf96da-1d3d-4265-8c9b-0d32024bdbed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:38.870206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bf96da-1d3d-4265-8c9b-0d32024bdbed admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:38.870492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bf96da-1d3d-4265-8c9b-0d32024bdbed admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:38.870770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bf96da-1d3d-4265-8c9b-0d32024bdbed admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:38.871516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bf96da-1d3d-4265-8c9b-0d32024bdbed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:38.873126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bf96da-1d3d-4265-8c9b-0d32024bdbed admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:38.884090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1257/2517] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:37:38.888162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07529dc-cb99-4c0c-9f99-3d6781101ef7 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:38.897999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e07529dc-cb99-4c0c-9f99-3d6781101ef7 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:38.902503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e67d96d-f699-4016-bb14-fdf1e63d546e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:38.903363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e67d96d-f699-4016-bb14-fdf1e63d546e admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:38.903650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e67d96d-f699-4016-bb14-fdf1e63d546e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:38.903956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e67d96d-f699-4016-bb14-fdf1e63d546e admin admin] PATH_INFO: `/v3/projects/6651ad0ba7744691968ec1deae3c9eca/users/9e438043c47d46f19efbc96decf17a5f/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:38.919265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e67d96d-f699-4016-bb14-fdf1e63d546e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6651ad0ba7744691968ec1deae3c9eca, user_id=9e438043c47d46f19efbc96decf17a5f, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:38.922644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e67d96d-f699-4016-bb14-fdf1e63d546e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:38.974874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1261/2518] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:37:38 2022] PUT /identity/v3/projects/6651ad0ba7744691968ec1deae3c9eca/users/9e438043c47d46f19efbc96decf17a5f/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 87 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:38.978750 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bf96da-1d3d-4265-8c9b-0d32024bdbed admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:38.988836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bf96da-1d3d-4265-8c9b-0d32024bdbed admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:39.044945 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78865ff-f9c2-41e4-bd91-3e19d1a534e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:39.045624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78865ff-f9c2-41e4-bd91-3e19d1a534e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:39.045908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78865ff-f9c2-41e4-bd91-3e19d1a534e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:39.046186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78865ff-f9c2-41e4-bd91-3e19d1a534e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:39.046900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78865ff-f9c2-41e4-bd91-3e19d1a534e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:39.048493 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78865ff-f9c2-41e4-bd91-3e19d1a534e8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:39.058038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1258/2519] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:38 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:39.062711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e67d96d-f699-4016-bb14-fdf1e63d546e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:39.073221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e67d96d-f699-4016-bb14-fdf1e63d546e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:39.078389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a227ed-4c36-4b44-bce1-fb4d8e5d345f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:39.079256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a227ed-4c36-4b44-bce1-fb4d8e5d345f admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:39.079541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a227ed-4c36-4b44-bce1-fb4d8e5d345f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:39.079822 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a227ed-4c36-4b44-bce1-fb4d8e5d345f admin admin] PATH_INFO: `/v3/projects/6651ad0ba7744691968ec1deae3c9eca/users/9e438043c47d46f19efbc96decf17a5f/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:39.081844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a227ed-4c36-4b44-bce1-fb4d8e5d345f admin admin] RBAC: Authorizing `identity:create_grant(project_id=6651ad0ba7744691968ec1deae3c9eca, user_id=9e438043c47d46f19efbc96decf17a5f, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:39.084513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a227ed-4c36-4b44-bce1-fb4d8e5d345f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:39.094485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1262/2520] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:37:39 2022] PUT /identity/v3/projects/6651ad0ba7744691968ec1deae3c9eca/users/9e438043c47d46f19efbc96decf17a5f/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:39.104569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7b64d6-1a13-4cff-b449-d22ba55942a6 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:39.104825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7b64d6-1a13-4cff-b449-d22ba55942a6 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:39.105110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7b64d6-1a13-4cff-b449-d22ba55942a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:39.131304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5e7b64d6-1a13-4cff-b449-d22ba55942a6 None None] MFA Rules not processed for user `9e438043c47d46f19efbc96decf17a5f`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:39.132491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5e7b64d6-1a13-4cff-b449-d22ba55942a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:39.141964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:39.199434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1259/2521] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:39 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:39.210997 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a227ed-4c36-4b44-bce1-fb4d8e5d345f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:39.212660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c7a227ed-4c36-4b44-bce1-fb4d8e5d345f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:39.268035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a227ed-4c36-4b44-bce1-fb4d8e5d345f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:39.271626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d19df437-d788-4438-89a9-4113159338bb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:39.272265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19df437-d788-4438-89a9-4113159338bb service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:39.272550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19df437-d788-4438-89a9-4113159338bb service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:39.272864 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19df437-d788-4438-89a9-4113159338bb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:39.281705 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19df437-d788-4438-89a9-4113159338bb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:39.283313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d19df437-d788-4438-89a9-4113159338bb service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:39.292546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1263/2522] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:39 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:37:40.174137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7b64d6-1a13-4cff-b449-d22ba55942a6 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:40.234355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7b64d6-1a13-4cff-b449-d22ba55942a6 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:40.238824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:40.239678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:40.239956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:40.240232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] PATH_INFO: `/v3/users/c59af2d25ae94ad1adfe0b9c67a59e19` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:40.241490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] RBAC: Authorizing `identity:delete_user(user_id=c59af2d25ae94ad1adfe0b9c67a59e19)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:40.243464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:40.388005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c59af2d25ae94ad1adfe0b9c67a59e19'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:40.388653 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c59af2d25ae94ad1adfe0b9c67a59e19'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:40.396034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c59af2d25ae94ad1adfe0b9c67a59e19'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:40.454459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c59af2d25ae94ad1adfe0b9c67a59e19'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:40.455080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c59af2d25ae94ad1adfe0b9c67a59e19'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:40.473774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1260/2523] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:37:40 2022] DELETE /identity/v3/users/c59af2d25ae94ad1adfe0b9c67a59e19 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:40.478335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c3c159-fc11-486c-ba67-90e8f234234e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:40.478810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c3c159-fc11-486c-ba67-90e8f234234e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:40.478935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c3c159-fc11-486c-ba67-90e8f234234e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:40.496480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-37c3c159-fc11-486c-ba67-90e8f234234e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:40.497718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-37c3c159-fc11-486c-ba67-90e8f234234e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:40.557969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1264/2524] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:40.568406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:40.569828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:40.600321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:40.626278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6db406-68d6-4a57-8e83-c8ebe37c29ec admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:40.629985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5fd69b-b8f1-457a-81a6-92251e0c002d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:40.630657 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5fd69b-b8f1-457a-81a6-92251e0c002d service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:40.631138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5fd69b-b8f1-457a-81a6-92251e0c002d service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:40.631587 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5fd69b-b8f1-457a-81a6-92251e0c002d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:40.641419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5fd69b-b8f1-457a-81a6-92251e0c002d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:40.650868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5fd69b-b8f1-457a-81a6-92251e0c002d service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:40.652065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1261/2525] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:37:41.033422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c3c159-fc11-486c-ba67-90e8f234234e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:41.034973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-37c3c159-fc11-486c-ba67-90e8f234234e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:41.048263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c3c159-fc11-486c-ba67-90e8f234234e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:41.053955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:41.059301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:41.059301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:41.059301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] PATH_INFO: `/v3/projects/e57542abc6bc4b5faf5a05f91f2d417e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:41.060168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] RBAC: Authorizing `identity:delete_project(project_id=e57542abc6bc4b5faf5a05f91f2d417e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:41.062629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:41.188936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:INSERT=5,keystone:DELETE=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:41.191831 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e57542abc6bc4b5faf5a05f91f2d417e', 'request_id': 'req-b1b22121-3656-44c1-9104-03ef3fd399ab'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.254580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e57542abc6bc4b5faf5a05f91f2d417e', 'request_id': 'req-b1b22121-3656-44c1-9104-03ef3fd399ab'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.260914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e57542abc6bc4b5faf5a05f91f2d417e', 'request_id': 'req-b1b22121-3656-44c1-9104-03ef3fd399ab'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.271456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] The token cache is being invalidate because project e57542abc6bc4b5faf5a05f91f2d417e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:37:41.271743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.272376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.274044 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1265/2526] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:37:41 2022] DELETE /identity/v3/projects/e57542abc6bc4b5faf5a05f91f2d417e => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:41.278027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5fd69b-b8f1-457a-81a6-92251e0c002d service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:41.288133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5fd69b-b8f1-457a-81a6-92251e0c002d service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:41.346304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:41.347178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:41.347462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:41.347758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] PATH_INFO: `/v3/users/9e438043c47d46f19efbc96decf17a5f` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:41.348988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9e438043c47d46f19efbc96decf17a5f)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:41.350927 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:41.489204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9e438043c47d46f19efbc96decf17a5f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.489936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9e438043c47d46f19efbc96decf17a5f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.496917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9e438043c47d46f19efbc96decf17a5f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.580437 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9e438043c47d46f19efbc96decf17a5f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.581029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-452936a5-6797-4b12-89fb-3e35735972d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9e438043c47d46f19efbc96decf17a5f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:41.598842 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1262/2527] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:37:41 2022] DELETE /identity/v3/users/9e438043c47d46f19efbc96decf17a5f => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:41.971023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:41.972481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:41.985525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1b22121-3656-44c1-9104-03ef3fd399ab admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:42.042874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:42.043730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.044015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.044291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] PATH_INFO: `/v3/projects/6651ad0ba7744691968ec1deae3c9eca` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.045511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] RBAC: Authorizing `identity:delete_project(project_id=6651ad0ba7744691968ec1deae3c9eca)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:42.047422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:42.162167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6651ad0ba7744691968ec1deae3c9eca', 'request_id': 'req-b558f2c6-c35d-4ef4-a44e-94372e55389b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:42.231433 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6651ad0ba7744691968ec1deae3c9eca', 'request_id': 'req-b558f2c6-c35d-4ef4-a44e-94372e55389b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:42.238374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6651ad0ba7744691968ec1deae3c9eca', 'request_id': 'req-b558f2c6-c35d-4ef4-a44e-94372e55389b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:42.250073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] The token cache is being invalidate because project 6651ad0ba7744691968ec1deae3c9eca was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:37:42.250419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:42.251145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-b558f2c6-c35d-4ef4-a44e-94372e55389b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:37:42.253254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1266/2528] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:37:41 2022] DELETE /identity/v3/projects/6651ad0ba7744691968ec1deae3c9eca => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:42.258283 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18684a9e-2bd4-4f43-b0f9-2816c19d6bff None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.258574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18684a9e-2bd4-4f43-b0f9-2816c19d6bff None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.258933 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18684a9e-2bd4-4f43-b0f9-2816c19d6bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.277705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-18684a9e-2bd4-4f43-b0f9-2816c19d6bff None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:42.278921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-18684a9e-2bd4-4f43-b0f9-2816c19d6bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:42.323926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:42.339345 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1263/2529] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:42.347612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b734f8a-885f-4c33-bf5c-401fcce6ace5 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.347939 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b734f8a-885f-4c33-bf5c-401fcce6ace5 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.348225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b734f8a-885f-4c33-bf5c-401fcce6ace5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.366769 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-8b734f8a-885f-4c33-bf5c-401fcce6ace5 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:42.367987 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8b734f8a-885f-4c33-bf5c-401fcce6ace5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:42.376431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1267/2530] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:42.380616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18684a9e-2bd4-4f43-b0f9-2816c19d6bff None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:42.390076 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18684a9e-2bd4-4f43-b0f9-2816c19d6bff None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:42.395035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0051607f-fd97-4393-b776-e669a2c25e1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:42.395674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0051607f-fd97-4393-b776-e669a2c25e1b admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.395952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0051607f-fd97-4393-b776-e669a2c25e1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.396224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0051607f-fd97-4393-b776-e669a2c25e1b admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.397002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0051607f-fd97-4393-b776-e669a2c25e1b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:37:42.402407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0051607f-fd97-4393-b776-e669a2c25e1b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:42.402407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0051607f-fd97-4393-b776-e669a2c25e1b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:42.410040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1264/2531] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:37:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:37:42.414201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b734f8a-885f-4c33-bf5c-401fcce6ace5 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:42.423917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b734f8a-885f-4c33-bf5c-401fcce6ace5 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:42.428567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c913e3-37d7-49e4-8d03-c188cee21ccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:42.429254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c913e3-37d7-49e4-8d03-c188cee21ccc admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.429533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c913e3-37d7-49e4-8d03-c188cee21ccc admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.429835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c913e3-37d7-49e4-8d03-c188cee21ccc admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.430512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c913e3-37d7-49e4-8d03-c188cee21ccc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:42.432401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c913e3-37d7-49e4-8d03-c188cee21ccc admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:42.495684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1268/2532] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:37:42 2022] POST /identity/v3/projects => generated 389 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:37:42.500237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0051607f-fd97-4393-b776-e669a2c25e1b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:42.510617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0051607f-fd97-4393-b776-e669a2c25e1b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:42.570412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284235d1-39d2-461a-b019-793f7bb84408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:42.574629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284235d1-39d2-461a-b019-793f7bb84408 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.574629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284235d1-39d2-461a-b019-793f7bb84408 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.574629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284235d1-39d2-461a-b019-793f7bb84408 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.574629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284235d1-39d2-461a-b019-793f7bb84408 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:42.574629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284235d1-39d2-461a-b019-793f7bb84408 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:42.689356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1265/2533] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:37:42 2022] POST /identity/v3/users => generated 295 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:37:42.693533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c913e3-37d7-49e4-8d03-c188cee21ccc admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:42.704373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c913e3-37d7-49e4-8d03-c188cee21ccc admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:42.708956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e49c6a0-233f-4a1e-953e-6e00a0a1e2dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:42.709611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e49c6a0-233f-4a1e-953e-6e00a0a1e2dd admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.709894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e49c6a0-233f-4a1e-953e-6e00a0a1e2dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.710190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e49c6a0-233f-4a1e-953e-6e00a0a1e2dd admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.710908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e49c6a0-233f-4a1e-953e-6e00a0a1e2dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:42.712509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e49c6a0-233f-4a1e-953e-6e00a0a1e2dd admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:42.722886 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1269/2534] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:42 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:42.726631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284235d1-39d2-461a-b019-793f7bb84408 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:42.736245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284235d1-39d2-461a-b019-793f7bb84408 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:42.740679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582e29c0-db6e-4993-b400-5bdab298d7e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:42.741349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582e29c0-db6e-4993-b400-5bdab298d7e2 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.741631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582e29c0-db6e-4993-b400-5bdab298d7e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.741909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582e29c0-db6e-4993-b400-5bdab298d7e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.742622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582e29c0-db6e-4993-b400-5bdab298d7e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:42.744202 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582e29c0-db6e-4993-b400-5bdab298d7e2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:42.754279 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1266/2535] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:42 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:42.757965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e49c6a0-233f-4a1e-953e-6e00a0a1e2dd admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:42.768149 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e49c6a0-233f-4a1e-953e-6e00a0a1e2dd admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:42.772589 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4abc9e6-a3a4-4ee6-a291-75e3e6c98f06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:42.773468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4abc9e6-a3a4-4ee6-a291-75e3e6c98f06 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.773749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4abc9e6-a3a4-4ee6-a291-75e3e6c98f06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.774022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4abc9e6-a3a4-4ee6-a291-75e3e6c98f06 admin admin] PATH_INFO: `/v3/projects/2786591284634c269e0ff0ac16fce1ac/users/88804d909de54f0b8fd28f2476986e60/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.788859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4abc9e6-a3a4-4ee6-a291-75e3e6c98f06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2786591284634c269e0ff0ac16fce1ac, user_id=88804d909de54f0b8fd28f2476986e60, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:42.791514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4abc9e6-a3a4-4ee6-a291-75e3e6c98f06 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:42.906784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:DELETE=11,keystone:SELECT=85,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:42.911468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1270/2536] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:37:42 2022] PUT /identity/v3/projects/2786591284634c269e0ff0ac16fce1ac/users/88804d909de54f0b8fd28f2476986e60/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:37:42.923371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f693931-c865-478f-a660-44d3e99a6a1a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:42.923656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f693931-c865-478f-a660-44d3e99a6a1a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:42.923956 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f693931-c865-478f-a660-44d3e99a6a1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:42.951458 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4f693931-c865-478f-a660-44d3e99a6a1a None None] MFA Rules not processed for user `88804d909de54f0b8fd28f2476986e60`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:42.952720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4f693931-c865-478f-a660-44d3e99a6a1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:43.016116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1267/2537] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:42 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:37:43.020555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4abc9e6-a3a4-4ee6-a291-75e3e6c98f06 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:43.082680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4abc9e6-a3a4-4ee6-a291-75e3e6c98f06 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:43.087478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0ddaf-4400-4726-925d-950f95cefb44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:43.089580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0ddaf-4400-4726-925d-950f95cefb44 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:43.089580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0ddaf-4400-4726-925d-950f95cefb44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:43.089580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0ddaf-4400-4726-925d-950f95cefb44 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:43.089580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab0ddaf-4400-4726-925d-950f95cefb44 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:43.091320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab0ddaf-4400-4726-925d-950f95cefb44 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:43.164849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1271/2538] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:37:43 2022] POST /identity/v3/projects => generated 391 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:37:43.168953 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f693931-c865-478f-a660-44d3e99a6a1a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:43.179751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f693931-c865-478f-a660-44d3e99a6a1a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:43.221616 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:43.235856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2cbf9-80c4-44ac-a166-cebe49a5c4a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:43.236520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2cbf9-80c4-44ac-a166-cebe49a5c4a1 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:43.236834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2cbf9-80c4-44ac-a166-cebe49a5c4a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:43.237117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2cbf9-80c4-44ac-a166-cebe49a5c4a1 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:43.237808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b2cbf9-80c4-44ac-a166-cebe49a5c4a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:43.239748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b2cbf9-80c4-44ac-a166-cebe49a5c4a1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:43.299630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1268/2539] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:37:43 2022] POST /identity/v3/users => generated 302 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:37:43.303747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0ddaf-4400-4726-925d-950f95cefb44 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:43.314087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0ddaf-4400-4726-925d-950f95cefb44 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:43.318962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a581b-3224-4abe-9d8c-6b8afeef71ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:43.319629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a581b-3224-4abe-9d8c-6b8afeef71ae admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:43.319920 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a581b-3224-4abe-9d8c-6b8afeef71ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:43.320204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487a581b-3224-4abe-9d8c-6b8afeef71ae admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:43.320975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487a581b-3224-4abe-9d8c-6b8afeef71ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:43.322641 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487a581b-3224-4abe-9d8c-6b8afeef71ae admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:43.336216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1272/2540] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:43.338214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2cbf9-80c4-44ac-a166-cebe49a5c4a1 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:43.349099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2cbf9-80c4-44ac-a166-cebe49a5c4a1 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:43.353746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7189f69f-4eca-4e05-b9f2-fd3fe2b6af40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:43.354624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7189f69f-4eca-4e05-b9f2-fd3fe2b6af40 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:43.354912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7189f69f-4eca-4e05-b9f2-fd3fe2b6af40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:43.355192 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7189f69f-4eca-4e05-b9f2-fd3fe2b6af40 admin admin] PATH_INFO: `/v3/projects/9573d34f9b49481eb95ae39a9b9b0865/users/856ff42dc9e24e299152dd0c1c5d161a/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:43.369838 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7189f69f-4eca-4e05-b9f2-fd3fe2b6af40 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9573d34f9b49481eb95ae39a9b9b0865, user_id=856ff42dc9e24e299152dd0c1c5d161a, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:43.372568 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7189f69f-4eca-4e05-b9f2-fd3fe2b6af40 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:43.426205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1269/2541] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:37:43 2022] PUT /identity/v3/projects/9573d34f9b49481eb95ae39a9b9b0865/users/856ff42dc9e24e299152dd0c1c5d161a/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:43.433944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a581b-3224-4abe-9d8c-6b8afeef71ae admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:43.444874 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487a581b-3224-4abe-9d8c-6b8afeef71ae admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:43.502639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7d1181-b7db-4a25-b911-25cdcfa038d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:43.503231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d1181-b7db-4a25-b911-25cdcfa038d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:43.503519 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d1181-b7db-4a25-b911-25cdcfa038d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:43.503803 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7d1181-b7db-4a25-b911-25cdcfa038d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:43.504527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7d1181-b7db-4a25-b911-25cdcfa038d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:43.506186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f7d1181-b7db-4a25-b911-25cdcfa038d4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:43.516425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1273/2542] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:37:43 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:37:43.520736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7189f69f-4eca-4e05-b9f2-fd3fe2b6af40 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:43.531054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7189f69f-4eca-4e05-b9f2-fd3fe2b6af40 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:43.535611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe57db23-33e6-41f2-a005-d5cdf7f85d4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:43.536516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe57db23-33e6-41f2-a005-d5cdf7f85d4e admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:43.536829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe57db23-33e6-41f2-a005-d5cdf7f85d4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:43.537112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe57db23-33e6-41f2-a005-d5cdf7f85d4e admin admin] PATH_INFO: `/v3/projects/9573d34f9b49481eb95ae39a9b9b0865/users/856ff42dc9e24e299152dd0c1c5d161a/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:43.539063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe57db23-33e6-41f2-a005-d5cdf7f85d4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9573d34f9b49481eb95ae39a9b9b0865, user_id=856ff42dc9e24e299152dd0c1c5d161a, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:43.541764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe57db23-33e6-41f2-a005-d5cdf7f85d4e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:43.551716 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1270/2543] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:37:43 2022] PUT /identity/v3/projects/9573d34f9b49481eb95ae39a9b9b0865/users/856ff42dc9e24e299152dd0c1c5d161a/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:37:43.562226 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670ee73b-fc2a-4a1e-9eba-c39dc4e6cf72 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:43.562518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670ee73b-fc2a-4a1e-9eba-c39dc4e6cf72 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:43.562800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-670ee73b-fc2a-4a1e-9eba-c39dc4e6cf72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:43.589657 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-670ee73b-fc2a-4a1e-9eba-c39dc4e6cf72 None None] MFA Rules not processed for user `856ff42dc9e24e299152dd0c1c5d161a`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:37:43.590923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-670ee73b-fc2a-4a1e-9eba-c39dc4e6cf72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:43.651012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:43.659635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1274/2544] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:37:43 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:37:43.671517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe57db23-33e6-41f2-a005-d5cdf7f85d4e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:43.673278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-fe57db23-33e6-41f2-a005-d5cdf7f85d4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:37:43.730031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe57db23-33e6-41f2-a005-d5cdf7f85d4e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:43.733668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d9c0eb-7fe4-4d01-93d6-7a8a24db5b35 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:43.734303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d9c0eb-7fe4-4d01-93d6-7a8a24db5b35 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:43.734590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d9c0eb-7fe4-4d01-93d6-7a8a24db5b35 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:43.734869 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d9c0eb-7fe4-4d01-93d6-7a8a24db5b35 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:43.743631 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d9c0eb-7fe4-4d01-93d6-7a8a24db5b35 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:43.745092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d9c0eb-7fe4-4d01-93d6-7a8a24db5b35 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:43.754802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1271/2545] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:43 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:37:43.799113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670ee73b-fc2a-4a1e-9eba-c39dc4e6cf72 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:37:43.808726 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-670ee73b-fc2a-4a1e-9eba-c39dc4e6cf72 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:37:43.813144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3604b717-833d-4752-94ac-64630a098c11 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:37:43.813784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3604b717-833d-4752-94ac-64630a098c11 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:37:43.814070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3604b717-833d-4752-94ac-64630a098c11 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:37:43.814352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3604b717-833d-4752-94ac-64630a098c11 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:37:43.870355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3604b717-833d-4752-94ac-64630a098c11 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:37:43.871786 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3604b717-833d-4752-94ac-64630a098c11 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:37:43.883931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1275/2546] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:37:43 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:37:53.750649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=50,keystone:INSERT=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:37:53.877151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=30 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:39:30.886054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea135b-0711-41ac-9e52-6ecaf17130f7 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:39:30.886936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea135b-0711-41ac-9e52-6ecaf17130f7 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:39:30.886936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ea135b-0711-41ac-9e52-6ecaf17130f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:39:30.914826 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-39ea135b-0711-41ac-9e52-6ecaf17130f7 None None] MFA Rules not processed for user `434ae422b68e4145ad8ecac27f450cbc`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:39:30.915989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-39ea135b-0711-41ac-9e52-6ecaf17130f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:30.986633 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1272/2547] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:39:30 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:39:31.275534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3604b717-833d-4752-94ac-64630a098c11 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:39:31.277007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3604b717-833d-4752-94ac-64630a098c11 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:31.342710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3604b717-833d-4752-94ac-64630a098c11 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:39:31.346794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:39:31.347434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:39:31.347717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:39:31.347993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:39:31.349757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:31.359477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:39:31.360919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:39:31.368668 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1276/2548] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:39:31 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:39:31.761929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea135b-0711-41ac-9e52-6ecaf17130f7 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:39:31.763311 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-39ea135b-0711-41ac-9e52-6ecaf17130f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:31.848471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ea135b-0711-41ac-9e52-6ecaf17130f7 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:39:31.853583 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:39:31.854228 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:39:31.854515 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:39:31.854799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:39:31.856570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:31.867666 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:39:31.869284 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:39:31.879090 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1273/2549] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:39:31 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:39:32.922115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:39:32.923535 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:32.938498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bce9e9-1972-4b61-95f1-920e309adfb3 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:39:32.943481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c50e33-1cad-4b1e-9398-c1c00db2a9c6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:39:32.944115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c50e33-1cad-4b1e-9398-c1c00db2a9c6 service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:39:32.944402 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c50e33-1cad-4b1e-9398-c1c00db2a9c6 service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:39:32.944685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c50e33-1cad-4b1e-9398-c1c00db2a9c6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:39:32.946429 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f5c50e33-1cad-4b1e-9398-c1c00db2a9c6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:32.957394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c50e33-1cad-4b1e-9398-c1c00db2a9c6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:39:32.958955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c50e33-1cad-4b1e-9398-c1c00db2a9c6 service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:39:32.969224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1277/2550] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:39:32 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:39:41.875329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=74 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:39:42.481186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:39:42.482919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:42.574822 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42992d9a-1606-46b9-b241-68b3e08a81c5 service placement] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:39:42.581424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a2f4c2-963e-4c00-9b6d-5ff795fd383b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:39:42.583830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a2f4c2-963e-4c00-9b6d-5ff795fd383b service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:39:42.583830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a2f4c2-963e-4c00-9b6d-5ff795fd383b service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:39:42.583830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a2f4c2-963e-4c00-9b6d-5ff795fd383b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:39:42.584678 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-38a2f4c2-963e-4c00-9b6d-5ff795fd383b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:42.597457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a2f4c2-963e-4c00-9b6d-5ff795fd383b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:39:42.599089 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a2f4c2-963e-4c00-9b6d-5ff795fd383b service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:39:42.609672 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1274/2551] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:39:42 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:39:42.964459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=36 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:39:46.896403 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c50e33-1cad-4b1e-9398-c1c00db2a9c6 service placement] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:39:46.909365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c50e33-1cad-4b1e-9398-c1c00db2a9c6 service placement] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:39:46.914921 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf656fc-04a0-4ff7-be71-c596d9ba44ab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:39:46.918646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf656fc-04a0-4ff7-be71-c596d9ba44ab service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:39:46.918646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf656fc-04a0-4ff7-be71-c596d9ba44ab service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:39:46.918646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf656fc-04a0-4ff7-be71-c596d9ba44ab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:39:46.918646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2cf656fc-04a0-4ff7-be71-c596d9ba44ab service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:39:46.931174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf656fc-04a0-4ff7-be71-c596d9ba44ab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:39:46.932817 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf656fc-04a0-4ff7-be71-c596d9ba44ab service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:39:46.944622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1278/2552] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:39:46 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:39:52.605217 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=41 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:39:56.939843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:22.604271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a2f4c2-963e-4c00-9b6d-5ff795fd383b service nova] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:22.664660 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a2f4c2-963e-4c00-9b6d-5ff795fd383b service nova] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:22.669416 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:22.670325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:22.670605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:22.670880 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] PATH_INFO: `/v3/users/88804d909de54f0b8fd28f2476986e60` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:22.672065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] RBAC: Authorizing `identity:delete_user(user_id=88804d909de54f0b8fd28f2476986e60)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:22.674520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:22.838893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88804d909de54f0b8fd28f2476986e60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:22.839578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88804d909de54f0b8fd28f2476986e60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:22.847902 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88804d909de54f0b8fd28f2476986e60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:22.899182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88804d909de54f0b8fd28f2476986e60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:22.899841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88804d909de54f0b8fd28f2476986e60'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:22.920096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1275/2553] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:40:22 2022] DELETE /identity/v3/users/88804d909de54f0b8fd28f2476986e60 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:22.934534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad112f88-1cda-403d-b45c-02b5e23d264e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:22.934534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad112f88-1cda-403d-b45c-02b5e23d264e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:22.934534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad112f88-1cda-403d-b45c-02b5e23d264e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:22.944099 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ad112f88-1cda-403d-b45c-02b5e23d264e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:22.945692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ad112f88-1cda-403d-b45c-02b5e23d264e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:23.006335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1279/2554] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:23.017040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:23.018426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:23.077455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f76366-e6f6-4138-9e02-4496ed534e9a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:23.090007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886f3a76-81e9-4b1e-9a79-25ca04b6abe1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:23.090790 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886f3a76-81e9-4b1e-9a79-25ca04b6abe1 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:23.091080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886f3a76-81e9-4b1e-9a79-25ca04b6abe1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:23.091358 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886f3a76-81e9-4b1e-9a79-25ca04b6abe1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:23.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886f3a76-81e9-4b1e-9a79-25ca04b6abe1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:23.102831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886f3a76-81e9-4b1e-9a79-25ca04b6abe1 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:23.112924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1276/2555] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:40:23.548521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad112f88-1cda-403d-b45c-02b5e23d264e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:23.552295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ad112f88-1cda-403d-b45c-02b5e23d264e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:23.571591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad112f88-1cda-403d-b45c-02b5e23d264e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:23.576485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:23.577720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:23.578358 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:23.579896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] PATH_INFO: `/v3/projects/2786591284634c269e0ff0ac16fce1ac` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:23.581587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2786591284634c269e0ff0ac16fce1ac)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:23.584912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:23.695136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2786591284634c269e0ff0ac16fce1ac', 'request_id': 'req-e8f55530-2534-4392-b0dd-9b8ea1c05f58'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:23.768630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2786591284634c269e0ff0ac16fce1ac', 'request_id': 'req-e8f55530-2534-4392-b0dd-9b8ea1c05f58'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:23.775396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2786591284634c269e0ff0ac16fce1ac', 'request_id': 'req-e8f55530-2534-4392-b0dd-9b8ea1c05f58'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:23.786782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] The token cache is being invalidate because project 2786591284634c269e0ff0ac16fce1ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:40:23.787075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:23.787733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:23.789318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1280/2556] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:40:23 2022] DELETE /identity/v3/projects/2786591284634c269e0ff0ac16fce1ac => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:23.793409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886f3a76-81e9-4b1e-9a79-25ca04b6abe1 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:23.803177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886f3a76-81e9-4b1e-9a79-25ca04b6abe1 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:23.821905 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:23.860176 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:23.861069 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:23.861351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:23.861624 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] PATH_INFO: `/v3/users/856ff42dc9e24e299152dd0c1c5d161a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:23.862799 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] RBAC: Authorizing `identity:delete_user(user_id=856ff42dc9e24e299152dd0c1c5d161a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:23.864729 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:24.027022 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '856ff42dc9e24e299152dd0c1c5d161a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.027779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '856ff42dc9e24e299152dd0c1c5d161a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.035424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '856ff42dc9e24e299152dd0c1c5d161a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.082087 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '856ff42dc9e24e299152dd0c1c5d161a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.082704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-1a646074-7d17-4bdc-9f9c-c392788cea9e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '856ff42dc9e24e299152dd0c1c5d161a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.099784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1277/2557] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:40:23 2022] DELETE /identity/v3/users/856ff42dc9e24e299152dd0c1c5d161a => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:24.478342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:24.479872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:24.495153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8f55530-2534-4392-b0dd-9b8ea1c05f58 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:24.555184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:24.556045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:24.556329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:24.556601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] PATH_INFO: `/v3/projects/9573d34f9b49481eb95ae39a9b9b0865` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:24.557836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] RBAC: Authorizing `identity:delete_project(project_id=9573d34f9b49481eb95ae39a9b9b0865)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:24.565818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:24.683698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:24.708789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9573d34f9b49481eb95ae39a9b9b0865', 'request_id': 'req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.787756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9573d34f9b49481eb95ae39a9b9b0865', 'request_id': 'req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.793032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9573d34f9b49481eb95ae39a9b9b0865', 'request_id': 'req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.806095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] The token cache is being invalidate because project 9573d34f9b49481eb95ae39a9b9b0865 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:40:24.806353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.807023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-6fe5e125-230e-400c-8f9c-1a6e9f60e9ed admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:24.808570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1281/2558] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:40:24 2022] DELETE /identity/v3/projects/9573d34f9b49481eb95ae39a9b9b0865 => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:24.813816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7689558-a057-4844-9431-c58b11c4c408 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:24.814311 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7689558-a057-4844-9431-c58b11c4c408 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:24.814409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7689558-a057-4844-9431-c58b11c4c408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:24.832836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-d7689558-a057-4844-9431-c58b11c4c408 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:24.834018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d7689558-a057-4844-9431-c58b11c4c408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:24.893001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1278/2559] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:24.900680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c099748-264c-4aa3-8fd7-063723704290 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:24.901221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c099748-264c-4aa3-8fd7-063723704290 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:24.901506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c099748-264c-4aa3-8fd7-063723704290 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:24.920991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5c099748-264c-4aa3-8fd7-063723704290 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:24.922357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5c099748-264c-4aa3-8fd7-063723704290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:24.930329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1282/2560] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:24.934627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7689558-a057-4844-9431-c58b11c4c408 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:24.948720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7689558-a057-4844-9431-c58b11c4c408 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:24.953693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e7d609-83a4-4637-85fe-7e4302071d4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:24.954889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e7d609-83a4-4637-85fe-7e4302071d4c admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:24.955219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e7d609-83a4-4637-85fe-7e4302071d4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:24.955496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e7d609-83a4-4637-85fe-7e4302071d4c admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:24.956229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e7d609-83a4-4637-85fe-7e4302071d4c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:40:24.956541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e7d609-83a4-4637-85fe-7e4302071d4c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:24.958196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e7d609-83a4-4637-85fe-7e4302071d4c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:24.970550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1279/2561] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:40:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:40:24.975070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c099748-264c-4aa3-8fd7-063723704290 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:24.984479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c099748-264c-4aa3-8fd7-063723704290 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:24.989230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f007527-40ce-4354-b346-4a40855407c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:24.989869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f007527-40ce-4354-b346-4a40855407c1 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:24.990158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f007527-40ce-4354-b346-4a40855407c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:24.990437 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f007527-40ce-4354-b346-4a40855407c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:24.991127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f007527-40ce-4354-b346-4a40855407c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:24.993059 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f007527-40ce-4354-b346-4a40855407c1 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:25.102332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1283/2562] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:40:24 2022] POST /identity/v3/projects => generated 389 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:25.106474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e7d609-83a4-4637-85fe-7e4302071d4c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:25.116363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e7d609-83a4-4637-85fe-7e4302071d4c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:25.142584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:25.174434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db0274d-20f5-4062-93ed-6a375c155dba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:25.175195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0274d-20f5-4062-93ed-6a375c155dba admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:25.175486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0274d-20f5-4062-93ed-6a375c155dba admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:25.175765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db0274d-20f5-4062-93ed-6a375c155dba admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:25.176501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db0274d-20f5-4062-93ed-6a375c155dba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:25.178486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db0274d-20f5-4062-93ed-6a375c155dba admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:25.254938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1280/2563] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:40:25 2022] POST /identity/v3/users => generated 295 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:25.259108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f007527-40ce-4354-b346-4a40855407c1 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:25.270493 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f007527-40ce-4354-b346-4a40855407c1 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:25.275124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d293f-b63d-4773-ba7f-01ccd61a171f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:25.275776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d293f-b63d-4773-ba7f-01ccd61a171f admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:25.276060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d293f-b63d-4773-ba7f-01ccd61a171f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:25.276365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-727d293f-b63d-4773-ba7f-01ccd61a171f admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:25.277124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727d293f-b63d-4773-ba7f-01ccd61a171f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:25.278760 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-727d293f-b63d-4773-ba7f-01ccd61a171f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:25.289206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1284/2564] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:25.293299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db0274d-20f5-4062-93ed-6a375c155dba admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:25.303111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db0274d-20f5-4062-93ed-6a375c155dba admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:25.307638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290e9e5d-37df-4ed2-9e8e-6734e319d512 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:25.308280 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e9e5d-37df-4ed2-9e8e-6734e319d512 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:25.308571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e9e5d-37df-4ed2-9e8e-6734e319d512 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:25.308885 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290e9e5d-37df-4ed2-9e8e-6734e319d512 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:25.309613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e9e5d-37df-4ed2-9e8e-6734e319d512 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:25.311475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290e9e5d-37df-4ed2-9e8e-6734e319d512 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:25.321999 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1281/2565] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:25.326365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d293f-b63d-4773-ba7f-01ccd61a171f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:25.337431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-727d293f-b63d-4773-ba7f-01ccd61a171f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:25.344495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28405af-55f2-449e-983e-4101bac282fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:25.344495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28405af-55f2-449e-983e-4101bac282fc admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:25.344495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28405af-55f2-449e-983e-4101bac282fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:25.344495 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28405af-55f2-449e-983e-4101bac282fc admin admin] PATH_INFO: `/v3/projects/3b2ccfe7d49d40d9aab3d416737a6f7e/users/163064cc5b704700b218144483dbfdaa/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:25.357902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28405af-55f2-449e-983e-4101bac282fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=3b2ccfe7d49d40d9aab3d416737a6f7e, user_id=163064cc5b704700b218144483dbfdaa, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:25.360556 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28405af-55f2-449e-983e-4101bac282fc admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:25.449620 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1285/2566] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:25 2022] PUT /identity/v3/projects/3b2ccfe7d49d40d9aab3d416737a6f7e/users/163064cc5b704700b218144483dbfdaa/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:25.460406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd0ae70-a5d3-48b2-942f-64b247f45199 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:25.460708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd0ae70-a5d3-48b2-942f-64b247f45199 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:25.461020 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dd0ae70-a5d3-48b2-942f-64b247f45199 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:25.490709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-9dd0ae70-a5d3-48b2-942f-64b247f45199 None None] MFA Rules not processed for user `163064cc5b704700b218144483dbfdaa`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:25.491910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9dd0ae70-a5d3-48b2-942f-64b247f45199 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:25.556735 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1282/2567] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:25 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:25.561573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28405af-55f2-449e-983e-4101bac282fc admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:25.623588 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28405af-55f2-449e-983e-4101bac282fc admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:25.628942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2cf3e4-713f-43fb-9c99-382c5430f0b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:25.629596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2cf3e4-713f-43fb-9c99-382c5430f0b3 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:25.629881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2cf3e4-713f-43fb-9c99-382c5430f0b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:25.630160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2cf3e4-713f-43fb-9c99-382c5430f0b3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:25.630855 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2cf3e4-713f-43fb-9c99-382c5430f0b3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:25.632757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2cf3e4-713f-43fb-9c99-382c5430f0b3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:25.698585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1286/2568] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:40:25 2022] POST /identity/v3/projects => generated 389 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:25.706957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd0ae70-a5d3-48b2-942f-64b247f45199 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:25.712870 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dd0ae70-a5d3-48b2-942f-64b247f45199 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:25.768184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2ac046-302b-46ce-910e-aac9a022376c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:25.768878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2ac046-302b-46ce-910e-aac9a022376c admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:25.769166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2ac046-302b-46ce-910e-aac9a022376c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:25.769446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2ac046-302b-46ce-910e-aac9a022376c admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:25.770219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2ac046-302b-46ce-910e-aac9a022376c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:25.772161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2ac046-302b-46ce-910e-aac9a022376c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:25.860027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1283/2569] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:40:25 2022] POST /identity/v3/users => generated 301 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:25.864400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2cf3e4-713f-43fb-9c99-382c5430f0b3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:25.874511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2cf3e4-713f-43fb-9c99-382c5430f0b3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:25.879890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d8b5af-7ab8-48d4-9e3a-f9f6581bd1d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:25.883998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d8b5af-7ab8-48d4-9e3a-f9f6581bd1d6 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:25.883998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d8b5af-7ab8-48d4-9e3a-f9f6581bd1d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:25.883998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d8b5af-7ab8-48d4-9e3a-f9f6581bd1d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:25.883998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d8b5af-7ab8-48d4-9e3a-f9f6581bd1d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:25.883998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d8b5af-7ab8-48d4-9e3a-f9f6581bd1d6 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:25.894462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1287/2570] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:25.898402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2ac046-302b-46ce-910e-aac9a022376c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:25.908937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2ac046-302b-46ce-910e-aac9a022376c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:25.913526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8c254a-5f15-4dc6-8e02-beb0a9078daf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:25.914517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8c254a-5f15-4dc6-8e02-beb0a9078daf admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:25.914789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8c254a-5f15-4dc6-8e02-beb0a9078daf admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:25.915061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8c254a-5f15-4dc6-8e02-beb0a9078daf admin admin] PATH_INFO: `/v3/projects/17a96b4b0966423ba0cb070625a2c276/users/c5599cb391854b508349225bc5e20383/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:25.930325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8c254a-5f15-4dc6-8e02-beb0a9078daf admin admin] RBAC: Authorizing `identity:create_grant(project_id=17a96b4b0966423ba0cb070625a2c276, user_id=c5599cb391854b508349225bc5e20383, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:25.933059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8c254a-5f15-4dc6-8e02-beb0a9078daf admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:26.025508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1284/2571] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:25 2022] PUT /identity/v3/projects/17a96b4b0966423ba0cb070625a2c276/users/c5599cb391854b508349225bc5e20383/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:40:26.029509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d8b5af-7ab8-48d4-9e3a-f9f6581bd1d6 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:26.040155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d8b5af-7ab8-48d4-9e3a-f9f6581bd1d6 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:26.060315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:26.097534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08452f8e-f159-4d57-8d50-328f7dce360b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:26.098196 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08452f8e-f159-4d57-8d50-328f7dce360b admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:26.098492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08452f8e-f159-4d57-8d50-328f7dce360b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:26.098774 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08452f8e-f159-4d57-8d50-328f7dce360b admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:26.099489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08452f8e-f159-4d57-8d50-328f7dce360b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:26.101142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08452f8e-f159-4d57-8d50-328f7dce360b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:26.111435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1288/2572] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:26 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:26.115547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8c254a-5f15-4dc6-8e02-beb0a9078daf admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:26.121159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:26.126446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8c254a-5f15-4dc6-8e02-beb0a9078daf admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:26.131118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467eef6a-dac7-420e-a343-5f0d3668af1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:26.135923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467eef6a-dac7-420e-a343-5f0d3668af1c admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:26.135923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467eef6a-dac7-420e-a343-5f0d3668af1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:26.135923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467eef6a-dac7-420e-a343-5f0d3668af1c admin admin] PATH_INFO: `/v3/projects/17a96b4b0966423ba0cb070625a2c276/users/c5599cb391854b508349225bc5e20383/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:26.135923 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467eef6a-dac7-420e-a343-5f0d3668af1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=17a96b4b0966423ba0cb070625a2c276, user_id=c5599cb391854b508349225bc5e20383, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:26.137277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467eef6a-dac7-420e-a343-5f0d3668af1c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:26.147401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1285/2573] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:26 2022] PUT /identity/v3/projects/17a96b4b0966423ba0cb070625a2c276/users/c5599cb391854b508349225bc5e20383/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:26.159070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63461da3-1d69-484b-bf1b-741e4235bdf3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:26.159356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63461da3-1d69-484b-bf1b-741e4235bdf3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:26.159635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63461da3-1d69-484b-bf1b-741e4235bdf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:26.186033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-63461da3-1d69-484b-bf1b-741e4235bdf3 None None] MFA Rules not processed for user `c5599cb391854b508349225bc5e20383`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:26.187253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-63461da3-1d69-484b-bf1b-741e4235bdf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:26.254395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1289/2574] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:26 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:26.264980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467eef6a-dac7-420e-a343-5f0d3668af1c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:26.267588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-467eef6a-dac7-420e-a343-5f0d3668af1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:26.324197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467eef6a-dac7-420e-a343-5f0d3668af1c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:26.327934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9d400c-dbc3-4e7e-9ce5-4f63fb421f01 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:26.336077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d400c-dbc3-4e7e-9ce5-4f63fb421f01 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:26.336871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d400c-dbc3-4e7e-9ce5-4f63fb421f01 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:26.336871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9d400c-dbc3-4e7e-9ce5-4f63fb421f01 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:26.346919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9d400c-dbc3-4e7e-9ce5-4f63fb421f01 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:26.348374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9d400c-dbc3-4e7e-9ce5-4f63fb421f01 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:26.357858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1286/2575] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:26 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:40:26.494877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63461da3-1d69-484b-bf1b-741e4235bdf3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:26.561546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63461da3-1d69-484b-bf1b-741e4235bdf3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:26.566259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:26.567180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:26.567463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:26.567748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] PATH_INFO: `/v3/users/163064cc5b704700b218144483dbfdaa` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:26.568998 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] RBAC: Authorizing `identity:delete_user(user_id=163064cc5b704700b218144483dbfdaa)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:26.571041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:26.758752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '163064cc5b704700b218144483dbfdaa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:26.759432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '163064cc5b704700b218144483dbfdaa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:26.760025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '163064cc5b704700b218144483dbfdaa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:26.761311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:26.767969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '163064cc5b704700b218144483dbfdaa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:26.841906 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '163064cc5b704700b218144483dbfdaa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:26.859347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1290/2576] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:40:26 2022] DELETE /identity/v3/users/163064cc5b704700b218144483dbfdaa => generated 0 bytes in 365 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:26.863825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab01449-d544-4040-bff2-0cc4481f0b70 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:26.864302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab01449-d544-4040-bff2-0cc4481f0b70 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:26.864492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab01449-d544-4040-bff2-0cc4481f0b70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:26.883966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6ab01449-d544-4040-bff2-0cc4481f0b70 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:26.883966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6ab01449-d544-4040-bff2-0cc4481f0b70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:26.942232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1287/2577] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:40:26.951987 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:26.953464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:27.013036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73b5c1e-5311-4cf2-8135-9c400943c948 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:27.016527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb34298-97cc-49f6-96e4-5bcfea2d743c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:27.017190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb34298-97cc-49f6-96e4-5bcfea2d743c service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:27.017474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb34298-97cc-49f6-96e4-5bcfea2d743c service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:27.017750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb34298-97cc-49f6-96e4-5bcfea2d743c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:27.027404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb34298-97cc-49f6-96e4-5bcfea2d743c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:27.028891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb34298-97cc-49f6-96e4-5bcfea2d743c service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:27.038978 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1291/2578] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:40:27.454993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab01449-d544-4040-bff2-0cc4481f0b70 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:27.456658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6ab01449-d544-4040-bff2-0cc4481f0b70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:27.468659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab01449-d544-4040-bff2-0cc4481f0b70 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:27.475334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:27.476356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:27.476797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:27.477184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] PATH_INFO: `/v3/projects/3b2ccfe7d49d40d9aab3d416737a6f7e` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:27.479121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=3b2ccfe7d49d40d9aab3d416737a6f7e)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:27.481298 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:27.588637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3b2ccfe7d49d40d9aab3d416737a6f7e', 'request_id': 'req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:27.595924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3b2ccfe7d49d40d9aab3d416737a6f7e', 'request_id': 'req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:27.601689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3b2ccfe7d49d40d9aab3d416737a6f7e', 'request_id': 'req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:27.698536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] The token cache is being invalidate because project 3b2ccfe7d49d40d9aab3d416737a6f7e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:40:27.698828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:27.699444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:27.701060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1288/2579] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:40:27 2022] DELETE /identity/v3/projects/3b2ccfe7d49d40d9aab3d416737a6f7e => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:27.705036 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb34298-97cc-49f6-96e4-5bcfea2d743c service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:27.714603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb34298-97cc-49f6-96e4-5bcfea2d743c service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:27.770899 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:27.771758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:27.772042 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:27.772320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] PATH_INFO: `/v3/users/c5599cb391854b508349225bc5e20383` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:27.773538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c5599cb391854b508349225bc5e20383)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:27.775458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:27.934897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c5599cb391854b508349225bc5e20383'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:27.935797 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c5599cb391854b508349225bc5e20383'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:27.936407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c5599cb391854b508349225bc5e20383'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:27.943467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c5599cb391854b508349225bc5e20383'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:28.002387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-fe648acd-176f-4836-8079-ebc5802431a7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c5599cb391854b508349225bc5e20383'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:28.007952 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:28.020563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1292/2580] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:40:27 2022] DELETE /identity/v3/users/c5599cb391854b508349225bc5e20383 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:28.458223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:28.461011 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:28.477215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3cc9e6-f7c0-4ac8-9625-723206aaaa8b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:28.490430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:28.535758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:28.536600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:28.536928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:28.537207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] PATH_INFO: `/v3/projects/17a96b4b0966423ba0cb070625a2c276` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:28.538581 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] RBAC: Authorizing `identity:delete_project(project_id=17a96b4b0966423ba0cb070625a2c276)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:28.540491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:28.655312 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '17a96b4b0966423ba0cb070625a2c276', 'request_id': 'req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:28.661990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '17a96b4b0966423ba0cb070625a2c276', 'request_id': 'req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:28.668115 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '17a96b4b0966423ba0cb070625a2c276', 'request_id': 'req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:28.744456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] The token cache is being invalidate because project 17a96b4b0966423ba0cb070625a2c276 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:40:28.744680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:28.745358 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-f9ecd71c-49ad-48c1-b07d-d3b68c715a73 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:28.746897 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1289/2581] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:40:28 2022] DELETE /identity/v3/projects/17a96b4b0966423ba0cb070625a2c276 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:28.751740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202afa0b-d761-4ced-8756-cdfeeafaedd9 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:28.752259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202afa0b-d761-4ced-8756-cdfeeafaedd9 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:28.752367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-202afa0b-d761-4ced-8756-cdfeeafaedd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:28.770622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-202afa0b-d761-4ced-8756-cdfeeafaedd9 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:28.771826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-202afa0b-d761-4ced-8756-cdfeeafaedd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:28.831272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1293/2582] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:40:28.839099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86918260-3cd1-4e12-a263-bf0d14453350 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:28.839400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86918260-3cd1-4e12-a263-bf0d14453350 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:28.839679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86918260-3cd1-4e12-a263-bf0d14453350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:28.860046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-86918260-3cd1-4e12-a263-bf0d14453350 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:28.861314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-86918260-3cd1-4e12-a263-bf0d14453350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:28.869152 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1290/2583] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:28.873298 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202afa0b-d761-4ced-8756-cdfeeafaedd9 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:28.883795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-202afa0b-d761-4ced-8756-cdfeeafaedd9 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:28.888494 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f269061-1b7e-4a1b-811f-c4b6c31679dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:28.889174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f269061-1b7e-4a1b-811f-c4b6c31679dc admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:28.889463 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f269061-1b7e-4a1b-811f-c4b6c31679dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:28.889748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f269061-1b7e-4a1b-811f-c4b6c31679dc admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:28.890468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f269061-1b7e-4a1b-811f-c4b6c31679dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:40:28.890776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f269061-1b7e-4a1b-811f-c4b6c31679dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:28.892374 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f269061-1b7e-4a1b-811f-c4b6c31679dc admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:28.904571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1294/2584] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:40:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:40:28.908860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86918260-3cd1-4e12-a263-bf0d14453350 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:28.918104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86918260-3cd1-4e12-a263-bf0d14453350 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:28.922793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c60bde-7f57-4fe1-a643-d9aa26b6cdda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:28.923443 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c60bde-7f57-4fe1-a643-d9aa26b6cdda admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:28.923763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c60bde-7f57-4fe1-a643-d9aa26b6cdda admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:28.924121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c60bde-7f57-4fe1-a643-d9aa26b6cdda admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:28.924872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c60bde-7f57-4fe1-a643-d9aa26b6cdda admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:28.927011 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c60bde-7f57-4fe1-a643-d9aa26b6cdda admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:29.010120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1291/2585] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:40:28 2022] POST /identity/v3/projects => generated 383 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:29.014207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f269061-1b7e-4a1b-811f-c4b6c31679dc admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:29.025049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f269061-1b7e-4a1b-811f-c4b6c31679dc admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:29.080781 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f2eabc-33a3-4e5f-85fe-a70718b7ee22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:29.081448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f2eabc-33a3-4e5f-85fe-a70718b7ee22 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:29.081735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f2eabc-33a3-4e5f-85fe-a70718b7ee22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:29.082016 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f2eabc-33a3-4e5f-85fe-a70718b7ee22 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:29.082708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f2eabc-33a3-4e5f-85fe-a70718b7ee22 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:29.084632 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24f2eabc-33a3-4e5f-85fe-a70718b7ee22 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:29.167951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1295/2586] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:40:29 2022] POST /identity/v3/users => generated 292 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:29.171895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c60bde-7f57-4fe1-a643-d9aa26b6cdda admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:29.182981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c60bde-7f57-4fe1-a643-d9aa26b6cdda admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:29.187933 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc556e3-b312-48ed-9328-3cafd4076e29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:29.188579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc556e3-b312-48ed-9328-3cafd4076e29 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:29.188936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc556e3-b312-48ed-9328-3cafd4076e29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:29.189260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc556e3-b312-48ed-9328-3cafd4076e29 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:29.189976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc556e3-b312-48ed-9328-3cafd4076e29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:29.191558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc556e3-b312-48ed-9328-3cafd4076e29 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:29.202125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1292/2587] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:29.206570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f2eabc-33a3-4e5f-85fe-a70718b7ee22 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:29.216407 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f2eabc-33a3-4e5f-85fe-a70718b7ee22 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:29.221560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30db3d6-409b-4b21-8078-368a476b1e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:29.222213 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30db3d6-409b-4b21-8078-368a476b1e84 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:29.222500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30db3d6-409b-4b21-8078-368a476b1e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:29.222780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30db3d6-409b-4b21-8078-368a476b1e84 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:29.223500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30db3d6-409b-4b21-8078-368a476b1e84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:29.225131 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30db3d6-409b-4b21-8078-368a476b1e84 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:29.235554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1296/2588] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:29 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:40:29.239659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc556e3-b312-48ed-9328-3cafd4076e29 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:29.249432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc556e3-b312-48ed-9328-3cafd4076e29 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:29.253921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459e7300-bd1a-424a-af8b-54885964639b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:29.254778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459e7300-bd1a-424a-af8b-54885964639b admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:29.255055 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459e7300-bd1a-424a-af8b-54885964639b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:29.255324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459e7300-bd1a-424a-af8b-54885964639b admin admin] PATH_INFO: `/v3/projects/90cdf5618b8e436bab9b99d8ff1c33e9/users/08024b64db024cbe94cb308004f729bc/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:29.269803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459e7300-bd1a-424a-af8b-54885964639b admin admin] RBAC: Authorizing `identity:create_grant(project_id=90cdf5618b8e436bab9b99d8ff1c33e9, user_id=08024b64db024cbe94cb308004f729bc, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:29.272379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459e7300-bd1a-424a-af8b-54885964639b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:29.362031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1293/2589] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:29 2022] PUT /identity/v3/projects/90cdf5618b8e436bab9b99d8ff1c33e9/users/08024b64db024cbe94cb308004f729bc/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:40:29.373512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddf0e0a-b0d2-4e64-bc28-d9df6ac17ff0 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:29.373791 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddf0e0a-b0d2-4e64-bc28-d9df6ac17ff0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:29.374071 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cddf0e0a-b0d2-4e64-bc28-d9df6ac17ff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:29.401112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-cddf0e0a-b0d2-4e64-bc28-d9df6ac17ff0 None None] MFA Rules not processed for user `08024b64db024cbe94cb308004f729bc`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:29.402317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-cddf0e0a-b0d2-4e64-bc28-d9df6ac17ff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:29.446958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:29.470072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1297/2590] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:29.475209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459e7300-bd1a-424a-af8b-54885964639b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:29.526112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:29.544965 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459e7300-bd1a-424a-af8b-54885964639b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:29.550100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8163b6aa-a867-4ecc-9099-1aedf9b7a5cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:29.550761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8163b6aa-a867-4ecc-9099-1aedf9b7a5cf admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:29.551047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8163b6aa-a867-4ecc-9099-1aedf9b7a5cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:29.551334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8163b6aa-a867-4ecc-9099-1aedf9b7a5cf admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:29.552025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8163b6aa-a867-4ecc-9099-1aedf9b7a5cf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:29.553984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8163b6aa-a867-4ecc-9099-1aedf9b7a5cf admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:29.684816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1294/2591] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:40:29 2022] POST /identity/v3/projects => generated 377 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:29.689096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddf0e0a-b0d2-4e64-bc28-d9df6ac17ff0 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:29.702124 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cddf0e0a-b0d2-4e64-bc28-d9df6ac17ff0 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:29.762584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5964b1ba-6c33-4315-9edc-d38378630a4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:29.763241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5964b1ba-6c33-4315-9edc-d38378630a4c admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:29.763521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5964b1ba-6c33-4315-9edc-d38378630a4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:29.763796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5964b1ba-6c33-4315-9edc-d38378630a4c admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:29.764465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5964b1ba-6c33-4315-9edc-d38378630a4c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:29.766408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5964b1ba-6c33-4315-9edc-d38378630a4c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:29.870457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1298/2592] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:40:29 2022] POST /identity/v3/users => generated 295 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:29.874436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8163b6aa-a867-4ecc-9099-1aedf9b7a5cf admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:29.885038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8163b6aa-a867-4ecc-9099-1aedf9b7a5cf admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:29.890302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f7d166-b2b5-48d6-b759-5216a838b9a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:29.891423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f7d166-b2b5-48d6-b759-5216a838b9a6 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:29.891713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f7d166-b2b5-48d6-b759-5216a838b9a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:29.891991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f7d166-b2b5-48d6-b759-5216a838b9a6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:29.893377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f7d166-b2b5-48d6-b759-5216a838b9a6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:29.895007 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f7d166-b2b5-48d6-b759-5216a838b9a6 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:29.905688 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1295/2593] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:29 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:29.909730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5964b1ba-6c33-4315-9edc-d38378630a4c admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:29.919735 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5964b1ba-6c33-4315-9edc-d38378630a4c admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:29.924744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce542f-bf3d-47cb-904a-c08d4afc2643 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:29.925644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce542f-bf3d-47cb-904a-c08d4afc2643 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:29.925937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce542f-bf3d-47cb-904a-c08d4afc2643 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:29.926216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce542f-bf3d-47cb-904a-c08d4afc2643 admin admin] PATH_INFO: `/v3/projects/210df7181e6944849d9a3ca10daf7d0a/users/3039f33f35dd40e88a7314317271fd38/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:29.941716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bce542f-bf3d-47cb-904a-c08d4afc2643 admin admin] RBAC: Authorizing `identity:create_grant(project_id=210df7181e6944849d9a3ca10daf7d0a, user_id=3039f33f35dd40e88a7314317271fd38, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:29.944387 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bce542f-bf3d-47cb-904a-c08d4afc2643 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:30.041944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1299/2594] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:29 2022] PUT /identity/v3/projects/210df7181e6944849d9a3ca10daf7d0a/users/3039f33f35dd40e88a7314317271fd38/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:30.046023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f7d166-b2b5-48d6-b759-5216a838b9a6 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:30.056327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f7d166-b2b5-48d6-b759-5216a838b9a6 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:30.110643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adb181e5-477c-4eeb-888f-2dbe9f918cd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:30.111801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb181e5-477c-4eeb-888f-2dbe9f918cd2 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:30.112085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb181e5-477c-4eeb-888f-2dbe9f918cd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:30.112362 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adb181e5-477c-4eeb-888f-2dbe9f918cd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:30.113113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb181e5-477c-4eeb-888f-2dbe9f918cd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:30.114712 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adb181e5-477c-4eeb-888f-2dbe9f918cd2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:30.124225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1296/2595] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:30 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:30.128956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce542f-bf3d-47cb-904a-c08d4afc2643 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:30.139140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce542f-bf3d-47cb-904a-c08d4afc2643 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:30.143739 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8557d86-106f-47cc-8b4f-cc0a33167cb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:30.144610 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8557d86-106f-47cc-8b4f-cc0a33167cb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:30.144935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8557d86-106f-47cc-8b4f-cc0a33167cb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:30.145219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8557d86-106f-47cc-8b4f-cc0a33167cb1 admin admin] PATH_INFO: `/v3/projects/210df7181e6944849d9a3ca10daf7d0a/users/3039f33f35dd40e88a7314317271fd38/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:30.147191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8557d86-106f-47cc-8b4f-cc0a33167cb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=210df7181e6944849d9a3ca10daf7d0a, user_id=3039f33f35dd40e88a7314317271fd38, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:30.149897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8557d86-106f-47cc-8b4f-cc0a33167cb1 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:30.159853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1300/2596] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:30 2022] PUT /identity/v3/projects/210df7181e6944849d9a3ca10daf7d0a/users/3039f33f35dd40e88a7314317271fd38/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:30.169869 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7729a-105f-4981-acaf-dc90a329aa86 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:30.170160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7729a-105f-4981-acaf-dc90a329aa86 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:30.170440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa7729a-105f-4981-acaf-dc90a329aa86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:30.198554 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-dfa7729a-105f-4981-acaf-dc90a329aa86 None None] MFA Rules not processed for user `3039f33f35dd40e88a7314317271fd38`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:30.199762 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-dfa7729a-105f-4981-acaf-dc90a329aa86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:30.265445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1297/2597] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:30 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:30.277162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8557d86-106f-47cc-8b4f-cc0a33167cb1 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:30.278845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e8557d86-106f-47cc-8b4f-cc0a33167cb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:30.336586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8557d86-106f-47cc-8b4f-cc0a33167cb1 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:30.340047 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe5dc9b-0f8a-405b-9239-a4a2c0849507 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:30.340694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe5dc9b-0f8a-405b-9239-a4a2c0849507 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:30.341011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe5dc9b-0f8a-405b-9239-a4a2c0849507 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:30.341292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fe5dc9b-0f8a-405b-9239-a4a2c0849507 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:30.350934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe5dc9b-0f8a-405b-9239-a4a2c0849507 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:30.352363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fe5dc9b-0f8a-405b-9239-a4a2c0849507 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:30.362523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1301/2598] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:30 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:40:30.517592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa7729a-105f-4981-acaf-dc90a329aa86 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:30.527136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa7729a-105f-4981-acaf-dc90a329aa86 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:30.530661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc328f3-e173-4c98-ad9a-d480e7431551 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:30.531290 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc328f3-e173-4c98-ad9a-d480e7431551 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:30.531573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc328f3-e173-4c98-ad9a-d480e7431551 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:30.531868 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc328f3-e173-4c98-ad9a-d480e7431551 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:30.565351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:30.587214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc328f3-e173-4c98-ad9a-d480e7431551 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:30.588649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc328f3-e173-4c98-ad9a-d480e7431551 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:30.598074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1298/2599] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:30 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:40:31.107903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe5dc9b-0f8a-405b-9239-a4a2c0849507 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:31.150110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:31.170738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fe5dc9b-0f8a-405b-9239-a4a2c0849507 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:31.175257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:31.176112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:31.176399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:31.176673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] PATH_INFO: `/v3/users/08024b64db024cbe94cb308004f729bc` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:31.177930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] RBAC: Authorizing `identity:delete_user(user_id=08024b64db024cbe94cb308004f729bc)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:31.179926 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:31.341827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '08024b64db024cbe94cb308004f729bc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:31.342477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '08024b64db024cbe94cb308004f729bc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:31.343126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '08024b64db024cbe94cb308004f729bc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:31.350234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '08024b64db024cbe94cb308004f729bc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:31.412780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '08024b64db024cbe94cb308004f729bc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:31.430502 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1302/2600] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:40:31 2022] DELETE /identity/v3/users/08024b64db024cbe94cb308004f729bc => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:31.435028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c1299-0c6f-481f-9757-c084ec4f64cc None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:31.435687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c1299-0c6f-481f-9757-c084ec4f64cc None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:31.435974 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4c1299-0c6f-481f-9757-c084ec4f64cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:31.458114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3d4c1299-0c6f-481f-9757-c084ec4f64cc None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:31.459342 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3d4c1299-0c6f-481f-9757-c084ec4f64cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:31.517407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1299/2601] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:31.526962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:31.528352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:31.584747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7bba27a-5586-4c04-b911-95353817d42f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:31.588424 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4599aa-b369-4820-abf7-2220eacaa0fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:31.589031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4599aa-b369-4820-abf7-2220eacaa0fb service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:31.589317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4599aa-b369-4820-abf7-2220eacaa0fb service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:31.589592 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e4599aa-b369-4820-abf7-2220eacaa0fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:31.598501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4599aa-b369-4820-abf7-2220eacaa0fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:31.599932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e4599aa-b369-4820-abf7-2220eacaa0fb service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:31.609180 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1303/2602] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:40:32.014757 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c1299-0c6f-481f-9757-c084ec4f64cc None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:32.016246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3d4c1299-0c6f-481f-9757-c084ec4f64cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:32.034853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4c1299-0c6f-481f-9757-c084ec4f64cc None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:32.049649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:32.049649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:32.049649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:32.049649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] PATH_INFO: `/v3/projects/90cdf5618b8e436bab9b99d8ff1c33e9` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:32.050489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=90cdf5618b8e436bab9b99d8ff1c33e9)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:32.050489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:32.168933 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '90cdf5618b8e436bab9b99d8ff1c33e9', 'request_id': 'req-45058165-e1d8-4ef1-bd87-8a3abee287c0'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.176116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '90cdf5618b8e436bab9b99d8ff1c33e9', 'request_id': 'req-45058165-e1d8-4ef1-bd87-8a3abee287c0'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.181766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '90cdf5618b8e436bab9b99d8ff1c33e9', 'request_id': 'req-45058165-e1d8-4ef1-bd87-8a3abee287c0'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.264388 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] The token cache is being invalidate because project 90cdf5618b8e436bab9b99d8ff1c33e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:40:32.264675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.265359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.266969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1300/2603] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:40:32 2022] DELETE /identity/v3/projects/90cdf5618b8e436bab9b99d8ff1c33e9 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:32.271324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4599aa-b369-4820-abf7-2220eacaa0fb service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:32.281170 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e4599aa-b369-4820-abf7-2220eacaa0fb service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:32.338032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:32.338910 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:32.339199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:32.339480 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] PATH_INFO: `/v3/users/3039f33f35dd40e88a7314317271fd38` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:32.340667 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3039f33f35dd40e88a7314317271fd38)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:32.342638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:32.344147 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:32.518892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3039f33f35dd40e88a7314317271fd38'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.519557 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3039f33f35dd40e88a7314317271fd38'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.520202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3039f33f35dd40e88a7314317271fd38'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.527119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3039f33f35dd40e88a7314317271fd38'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.596239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d2ac6884-71b1-481d-bd60-466568cd7f03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3039f33f35dd40e88a7314317271fd38'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:32.613884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1304/2604] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:40:32 2022] DELETE /identity/v3/users/3039f33f35dd40e88a7314317271fd38 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:33.111856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:33.113371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:33.126100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45058165-e1d8-4ef1-bd87-8a3abee287c0 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:33.207286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:33.208871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:33.209157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:33.209434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] PATH_INFO: `/v3/projects/210df7181e6944849d9a3ca10daf7d0a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:33.210689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=210df7181e6944849d9a3ca10daf7d0a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:33.212681 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:33.220353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:33.341966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '210df7181e6944849d9a3ca10daf7d0a', 'request_id': 'req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:33.357151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '210df7181e6944849d9a3ca10daf7d0a', 'request_id': 'req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:33.362734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '210df7181e6944849d9a3ca10daf7d0a', 'request_id': 'req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:33.449157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] The token cache is being invalidate because project 210df7181e6944849d9a3ca10daf7d0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:40:33.449552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:33.450140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0b4251c2-7493-4b1f-b144-fb8b0f8beb52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:33.451846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1301/2605] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:40:33 2022] DELETE /identity/v3/projects/210df7181e6944849d9a3ca10daf7d0a => generated 0 bytes in 340 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:33.460565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c4ab8a-58e0-4ebb-9de7-38b213622e6e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:33.461655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c4ab8a-58e0-4ebb-9de7-38b213622e6e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:33.462281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13c4ab8a-58e0-4ebb-9de7-38b213622e6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:33.482733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-13c4ab8a-58e0-4ebb-9de7-38b213622e6e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:33.485230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-13c4ab8a-58e0-4ebb-9de7-38b213622e6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:33.559301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1305/2606] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:33.567814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7468a2d0-f33b-45d4-92b7-c3f70dac88fa None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:33.568108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7468a2d0-f33b-45d4-92b7-c3f70dac88fa None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:33.568447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7468a2d0-f33b-45d4-92b7-c3f70dac88fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:33.589711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-7468a2d0-f33b-45d4-92b7-c3f70dac88fa None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:33.590922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7468a2d0-f33b-45d4-92b7-c3f70dac88fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:33.600129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1302/2607] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:40:33.604522 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c4ab8a-58e0-4ebb-9de7-38b213622e6e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:33.615108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13c4ab8a-58e0-4ebb-9de7-38b213622e6e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:33.619610 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e5e604-662e-4f0f-8093-7150566f3c45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:33.622824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e5e604-662e-4f0f-8093-7150566f3c45 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:33.622824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e5e604-662e-4f0f-8093-7150566f3c45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:33.622824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e5e604-662e-4f0f-8093-7150566f3c45 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:33.622824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e5e604-662e-4f0f-8093-7150566f3c45 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:40:33.622824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e5e604-662e-4f0f-8093-7150566f3c45 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:33.623527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76e5e604-662e-4f0f-8093-7150566f3c45 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:33.635219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1306/2608] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:40:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:40:33.639503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7468a2d0-f33b-45d4-92b7-c3f70dac88fa None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:33.649629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7468a2d0-f33b-45d4-92b7-c3f70dac88fa None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:33.654322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c571ca-9207-4d87-ada5-ebde6d7ab22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:33.660218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c571ca-9207-4d87-ada5-ebde6d7ab22d admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:33.660218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c571ca-9207-4d87-ada5-ebde6d7ab22d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:33.660218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c571ca-9207-4d87-ada5-ebde6d7ab22d admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:33.660218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c571ca-9207-4d87-ada5-ebde6d7ab22d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:33.660218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c571ca-9207-4d87-ada5-ebde6d7ab22d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:33.792509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1303/2609] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:40:33 2022] POST /identity/v3/projects => generated 377 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:33.796730 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e5e604-662e-4f0f-8093-7150566f3c45 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:33.808112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e5e604-662e-4f0f-8093-7150566f3c45 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:33.867404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74a8f66-df40-41d9-a12c-9747a346a6e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:33.868003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74a8f66-df40-41d9-a12c-9747a346a6e2 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:33.868289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74a8f66-df40-41d9-a12c-9747a346a6e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:33.868566 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74a8f66-df40-41d9-a12c-9747a346a6e2 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:33.869457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74a8f66-df40-41d9-a12c-9747a346a6e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:33.871491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f74a8f66-df40-41d9-a12c-9747a346a6e2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:33.881835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:33.987965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1307/2610] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:40:33 2022] POST /identity/v3/users => generated 289 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:33.995796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c571ca-9207-4d87-ada5-ebde6d7ab22d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:34.007608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c571ca-9207-4d87-ada5-ebde6d7ab22d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:34.012920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acdfaa6-63e5-47fe-8b7b-89e3fa0a30bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:34.013589 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acdfaa6-63e5-47fe-8b7b-89e3fa0a30bd admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:34.013876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acdfaa6-63e5-47fe-8b7b-89e3fa0a30bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:34.014156 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acdfaa6-63e5-47fe-8b7b-89e3fa0a30bd admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:34.014898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acdfaa6-63e5-47fe-8b7b-89e3fa0a30bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:34.016509 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4acdfaa6-63e5-47fe-8b7b-89e3fa0a30bd admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:34.028272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1304/2611] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:33 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:34.031787 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74a8f66-df40-41d9-a12c-9747a346a6e2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:34.042498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f74a8f66-df40-41d9-a12c-9747a346a6e2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:34.047265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a7be98-a009-4df2-9baf-2b0465292004 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:34.047929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a7be98-a009-4df2-9baf-2b0465292004 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:34.048219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a7be98-a009-4df2-9baf-2b0465292004 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:34.048518 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a7be98-a009-4df2-9baf-2b0465292004 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:34.049451 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a7be98-a009-4df2-9baf-2b0465292004 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:34.051484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a7be98-a009-4df2-9baf-2b0465292004 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:34.064397 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1308/2612] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:34 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:34.069165 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acdfaa6-63e5-47fe-8b7b-89e3fa0a30bd admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:34.079861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acdfaa6-63e5-47fe-8b7b-89e3fa0a30bd admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:34.085232 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef01e33-e38d-47b9-9ae6-b632177842e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:34.086107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef01e33-e38d-47b9-9ae6-b632177842e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:34.086386 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef01e33-e38d-47b9-9ae6-b632177842e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:34.086664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef01e33-e38d-47b9-9ae6-b632177842e9 admin admin] PATH_INFO: `/v3/projects/111fee5c9a9f4f2984bcbee87576f16d/users/3500cd37f2fc41e9bab2f96289b1a737/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:34.104613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef01e33-e38d-47b9-9ae6-b632177842e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=111fee5c9a9f4f2984bcbee87576f16d, user_id=3500cd37f2fc41e9bab2f96289b1a737, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:34.104613 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef01e33-e38d-47b9-9ae6-b632177842e9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:34.202658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1305/2613] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:34 2022] PUT /identity/v3/projects/111fee5c9a9f4f2984bcbee87576f16d/users/3500cd37f2fc41e9bab2f96289b1a737/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:34.214866 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331adb7a-e5c7-4555-8797-adb2a580dc5a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:34.215191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331adb7a-e5c7-4555-8797-adb2a580dc5a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:34.215442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331adb7a-e5c7-4555-8797-adb2a580dc5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:34.241895 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-331adb7a-e5c7-4555-8797-adb2a580dc5a None None] MFA Rules not processed for user `3500cd37f2fc41e9bab2f96289b1a737`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:34.243146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-331adb7a-e5c7-4555-8797-adb2a580dc5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:34.309418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1309/2614] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:34 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:34.323948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef01e33-e38d-47b9-9ae6-b632177842e9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:34.325871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6ef01e33-e38d-47b9-9ae6-b632177842e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:34.406969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef01e33-e38d-47b9-9ae6-b632177842e9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:34.411378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eec23f-7cf7-4e7b-bf55-fa88bcf879d8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a83b4eb626343f3aa5b1c9f3fc828fd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:34.412063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eec23f-7cf7-4e7b-bf55-fa88bcf879d8 service glance] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:34.412371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eec23f-7cf7-4e7b-bf55-fa88bcf879d8 service glance] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:34.412646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33eec23f-7cf7-4e7b-bf55-fa88bcf879d8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:34.421018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33eec23f-7cf7-4e7b-bf55-fa88bcf879d8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:34.422488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33eec23f-7cf7-4e7b-bf55-fa88bcf879d8 service glance] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:34.433341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1306/2615] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:34 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:40:34.554284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331adb7a-e5c7-4555-8797-adb2a580dc5a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:34.556154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-331adb7a-e5c7-4555-8797-adb2a580dc5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:34.617113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331adb7a-e5c7-4555-8797-adb2a580dc5a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:34.620862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db4ecf9-6300-41a7-9ff4-4e1ac12ed5d6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:34.621683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db4ecf9-6300-41a7-9ff4-4e1ac12ed5d6 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:34.622030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db4ecf9-6300-41a7-9ff4-4e1ac12ed5d6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:34.622373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db4ecf9-6300-41a7-9ff4-4e1ac12ed5d6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:34.624360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4db4ecf9-6300-41a7-9ff4-4e1ac12ed5d6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:34.694678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db4ecf9-6300-41a7-9ff4-4e1ac12ed5d6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:34.696113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db4ecf9-6300-41a7-9ff4-4e1ac12ed5d6 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:34.705843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1310/2616] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:40:35.357680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eec23f-7cf7-4e7b-bf55-fa88bcf879d8 service glance] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:35.359326 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-33eec23f-7cf7-4e7b-bf55-fa88bcf879d8 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:35.378928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33eec23f-7cf7-4e7b-bf55-fa88bcf879d8 service glance] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:35.383884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:35.384850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:35.385132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:35.385410 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] PATH_INFO: `/v3/users/849e36f8653d4d1f9a7fb1df8fb913a9` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:35.386591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=849e36f8653d4d1f9a7fb1df8fb913a9)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:35.388552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:35.391237 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:35.610328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '849e36f8653d4d1f9a7fb1df8fb913a9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:35.611084 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '849e36f8653d4d1f9a7fb1df8fb913a9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:35.618632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '849e36f8653d4d1f9a7fb1df8fb913a9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:35.699035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '849e36f8653d4d1f9a7fb1df8fb913a9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:35.699665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-10b22e05-c575-417c-b5ac-8ce0c3a801b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '849e36f8653d4d1f9a7fb1df8fb913a9'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:35.717322 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1307/2617] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:40:35 2022] DELETE /identity/v3/users/849e36f8653d4d1f9a7fb1df8fb913a9 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:35.994727 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db4ecf9-6300-41a7-9ff4-4e1ac12ed5d6 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:36.009281 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db4ecf9-6300-41a7-9ff4-4e1ac12ed5d6 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:36.034428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:36.077136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:36.079144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.079144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.079144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] PATH_INFO: `/v3/projects/ff2616337ae440a89047fd9a50218387` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.080476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ff2616337ae440a89047fd9a50218387)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:36.082617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:36.193660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ff2616337ae440a89047fd9a50218387', 'request_id': 'req-2782ebfd-3843-4bf6-83a3-6b840575eed7'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:36.264359 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ff2616337ae440a89047fd9a50218387', 'request_id': 'req-2782ebfd-3843-4bf6-83a3-6b840575eed7'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:36.271732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ff2616337ae440a89047fd9a50218387', 'request_id': 'req-2782ebfd-3843-4bf6-83a3-6b840575eed7'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:36.282931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] The token cache is being invalidate because project ff2616337ae440a89047fd9a50218387 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:40:36.283230 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:36.284483 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-2782ebfd-3843-4bf6-83a3-6b840575eed7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:36.286600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1311/2618] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:40:35 2022] DELETE /identity/v3/projects/ff2616337ae440a89047fd9a50218387 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:36.293171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e57011-2627-426e-8502-6c5d413be188 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.293532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e57011-2627-426e-8502-6c5d413be188 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.293779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e57011-2627-426e-8502-6c5d413be188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.316092 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-83e57011-2627-426e-8502-6c5d413be188 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:36.317383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-83e57011-2627-426e-8502-6c5d413be188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:36.379939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1308/2619] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:36.387690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e5f645-1bfa-4c4c-b057-1487e2d52e0d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.387981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e5f645-1bfa-4c4c-b057-1487e2d52e0d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.388301 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e5f645-1bfa-4c4c-b057-1487e2d52e0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.406678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-81e5f645-1bfa-4c4c-b057-1487e2d52e0d None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:36.407914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-81e5f645-1bfa-4c4c-b057-1487e2d52e0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:36.417095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1312/2620] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:36.421329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e57011-2627-426e-8502-6c5d413be188 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:36.431636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e57011-2627-426e-8502-6c5d413be188 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:36.436617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c6997-58be-44fd-b0c5-a16c77a3c937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:36.437323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c6997-58be-44fd-b0c5-a16c77a3c937 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.437635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c6997-58be-44fd-b0c5-a16c77a3c937 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.437915 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73c6997-58be-44fd-b0c5-a16c77a3c937 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.438675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c6997-58be-44fd-b0c5-a16c77a3c937 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:40:36.438985 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c6997-58be-44fd-b0c5-a16c77a3c937 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:36.441332 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73c6997-58be-44fd-b0c5-a16c77a3c937 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:36.456110 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1309/2621] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:40:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:40:36.461232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e5f645-1bfa-4c4c-b057-1487e2d52e0d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:36.471928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e5f645-1bfa-4c4c-b057-1487e2d52e0d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:36.476736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421723b-f8be-40c3-a04c-4be1df722833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:36.477432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421723b-f8be-40c3-a04c-4be1df722833 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.477712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421723b-f8be-40c3-a04c-4be1df722833 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.477993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d421723b-f8be-40c3-a04c-4be1df722833 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.478695 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d421723b-f8be-40c3-a04c-4be1df722833 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:36.480614 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d421723b-f8be-40c3-a04c-4be1df722833 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:36.592389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1313/2622] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:40:36 2022] POST /identity/v3/projects => generated 379 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:36.596617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c6997-58be-44fd-b0c5-a16c77a3c937 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:36.607159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73c6997-58be-44fd-b0c5-a16c77a3c937 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:36.666484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208ea7b3-51b5-4354-a446-79d6152c9a79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:36.667148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208ea7b3-51b5-4354-a446-79d6152c9a79 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.667431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208ea7b3-51b5-4354-a446-79d6152c9a79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.667770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208ea7b3-51b5-4354-a446-79d6152c9a79 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.668448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208ea7b3-51b5-4354-a446-79d6152c9a79 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:36.670385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208ea7b3-51b5-4354-a446-79d6152c9a79 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:36.750630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:36.757060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1310/2623] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:40:36 2022] POST /identity/v3/users => generated 290 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:36.761009 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421723b-f8be-40c3-a04c-4be1df722833 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:36.774104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d421723b-f8be-40c3-a04c-4be1df722833 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:36.778924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d218c0-27d9-4777-b875-544d43f785d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:36.779599 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d218c0-27d9-4777-b875-544d43f785d5 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.779884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d218c0-27d9-4777-b875-544d43f785d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.782545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d218c0-27d9-4777-b875-544d43f785d5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.782545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d218c0-27d9-4777-b875-544d43f785d5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:36.782545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d218c0-27d9-4777-b875-544d43f785d5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:36.794172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1314/2624] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:36.798169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208ea7b3-51b5-4354-a446-79d6152c9a79 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:36.807979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208ea7b3-51b5-4354-a446-79d6152c9a79 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:36.812889 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f189dcbf-a2ee-4972-9a83-cf5f95291a58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:36.813486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189dcbf-a2ee-4972-9a83-cf5f95291a58 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.813766 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189dcbf-a2ee-4972-9a83-cf5f95291a58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.814040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f189dcbf-a2ee-4972-9a83-cf5f95291a58 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.814811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f189dcbf-a2ee-4972-9a83-cf5f95291a58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:36.816395 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f189dcbf-a2ee-4972-9a83-cf5f95291a58 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:36.826861 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1311/2625] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:36.831538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d218c0-27d9-4777-b875-544d43f785d5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:36.842573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d218c0-27d9-4777-b875-544d43f785d5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:36.848519 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea54cb8-a90b-4e5c-89ed-850871772746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:36.849511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea54cb8-a90b-4e5c-89ed-850871772746 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.849893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea54cb8-a90b-4e5c-89ed-850871772746 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.850190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea54cb8-a90b-4e5c-89ed-850871772746 admin admin] PATH_INFO: `/v3/projects/372f8b4d0d5c4046ad91bea6d709a6ef/users/bf961149e78d4c0c8acb58f1cec447bd/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.866223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea54cb8-a90b-4e5c-89ed-850871772746 admin admin] RBAC: Authorizing `identity:create_grant(project_id=372f8b4d0d5c4046ad91bea6d709a6ef, user_id=bf961149e78d4c0c8acb58f1cec447bd, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:36.869001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cea54cb8-a90b-4e5c-89ed-850871772746 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:36.962426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1315/2626] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:36 2022] PUT /identity/v3/projects/372f8b4d0d5c4046ad91bea6d709a6ef/users/bf961149e78d4c0c8acb58f1cec447bd/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:36.967369 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1df798-0a76-4bae-9447-bfd506d58a21 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:36.967659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1df798-0a76-4bae-9447-bfd506d58a21 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:36.967937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1df798-0a76-4bae-9447-bfd506d58a21 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:36.989540 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-3c1df798-0a76-4bae-9447-bfd506d58a21 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:36.990800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-3c1df798-0a76-4bae-9447-bfd506d58a21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:37.057341 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1312/2627] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:37.065987 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea54cb8-a90b-4e5c-89ed-850871772746 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:37.066722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-cea54cb8-a90b-4e5c-89ed-850871772746 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:37.144186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cea54cb8-a90b-4e5c-89ed-850871772746 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:37.148944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac12e089-ada4-4899-b4f0-148ab30d90ee service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:37.149942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac12e089-ada4-4899-b4f0-148ab30d90ee service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:37.150292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac12e089-ada4-4899-b4f0-148ab30d90ee service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:37.150854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac12e089-ada4-4899-b4f0-148ab30d90ee service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:37.159659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:37.165894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac12e089-ada4-4899-b4f0-148ab30d90ee service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:37.174798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac12e089-ada4-4899-b4f0-148ab30d90ee service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:37.177777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1316/2628] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:40:42.948246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7006c3-e9b0-4197-95ce-df3f96c74813 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:42.948827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7006c3-e9b0-4197-95ce-df3f96c74813 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:42.948943 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7006c3-e9b0-4197-95ce-df3f96c74813 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:42.984473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5e7006c3-e9b0-4197-95ce-df3f96c74813 None None] MFA Rules not processed for user `bf961149e78d4c0c8acb58f1cec447bd`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:42.985749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5e7006c3-e9b0-4197-95ce-df3f96c74813 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:43.081146 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1313/2629] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:42 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:40:43.087554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac12e089-ada4-4899-b4f0-148ab30d90ee service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:43.105057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac12e089-ada4-4899-b4f0-148ab30d90ee service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:43.112673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd015d0-8c24-4e05-a89c-8b2b10797a89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:43.113624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd015d0-8c24-4e05-a89c-8b2b10797a89 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:43.114145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd015d0-8c24-4e05-a89c-8b2b10797a89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:43.114642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd015d0-8c24-4e05-a89c-8b2b10797a89 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:43.116109 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd015d0-8c24-4e05-a89c-8b2b10797a89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:43.119118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd015d0-8c24-4e05-a89c-8b2b10797a89 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:43.196360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1317/2630] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:40:43 2022] POST /identity/v3/projects => generated 379 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:43.200400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7006c3-e9b0-4197-95ce-df3f96c74813 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:43.215105 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7006c3-e9b0-4197-95ce-df3f96c74813 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:43.293979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c48a78-9e49-45e9-9302-dccc68117786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:43.294813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c48a78-9e49-45e9-9302-dccc68117786 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:43.295119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c48a78-9e49-45e9-9302-dccc68117786 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:43.295424 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c48a78-9e49-45e9-9302-dccc68117786 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:43.296270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c48a78-9e49-45e9-9302-dccc68117786 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:43.307005 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4c48a78-9e49-45e9-9302-dccc68117786 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:43.385438 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1314/2631] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:40:43 2022] POST /identity/v3/users => generated 296 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:40:43.389355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd015d0-8c24-4e05-a89c-8b2b10797a89 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:43.400486 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd015d0-8c24-4e05-a89c-8b2b10797a89 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:43.405747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f06bbb1-f658-4071-881b-ac7d1da01d37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:43.406517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f06bbb1-f658-4071-881b-ac7d1da01d37 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:43.406815 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f06bbb1-f658-4071-881b-ac7d1da01d37 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:43.407092 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f06bbb1-f658-4071-881b-ac7d1da01d37 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:43.408604 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f06bbb1-f658-4071-881b-ac7d1da01d37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:43.410385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f06bbb1-f658-4071-881b-ac7d1da01d37 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:43.422413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1318/2632] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:43.427224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c48a78-9e49-45e9-9302-dccc68117786 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:43.443411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c48a78-9e49-45e9-9302-dccc68117786 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:43.446523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b9b661-c588-487d-a707-512e8ed1bc15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:43.447502 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b9b661-c588-487d-a707-512e8ed1bc15 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:43.447882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b9b661-c588-487d-a707-512e8ed1bc15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:43.448486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b9b661-c588-487d-a707-512e8ed1bc15 admin admin] PATH_INFO: `/v3/projects/02e2931fabaf4f15960727403906fd80/users/e4d5b2f941f8496e95f7493c8fb0252a/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:43.451354 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:43.469363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b9b661-c588-487d-a707-512e8ed1bc15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=02e2931fabaf4f15960727403906fd80, user_id=e4d5b2f941f8496e95f7493c8fb0252a, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:43.473225 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88b9b661-c588-487d-a707-512e8ed1bc15 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:43.553157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1315/2633] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:43 2022] PUT /identity/v3/projects/02e2931fabaf4f15960727403906fd80/users/e4d5b2f941f8496e95f7493c8fb0252a/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:43.560331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f06bbb1-f658-4071-881b-ac7d1da01d37 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:43.571993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f06bbb1-f658-4071-881b-ac7d1da01d37 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:43.633065 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04dca2f3-3684-402d-ab80-fe3f0b8eb371 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:43.633741 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dca2f3-3684-402d-ab80-fe3f0b8eb371 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:43.634025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dca2f3-3684-402d-ab80-fe3f0b8eb371 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:43.634360 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04dca2f3-3684-402d-ab80-fe3f0b8eb371 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:43.635068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dca2f3-3684-402d-ab80-fe3f0b8eb371 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:43.636681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04dca2f3-3684-402d-ab80-fe3f0b8eb371 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:43.648225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1319/2634] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:40:43 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:40:43.653158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b9b661-c588-487d-a707-512e8ed1bc15 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:43.676319 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b9b661-c588-487d-a707-512e8ed1bc15 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:43.691491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8118fd15-9616-4667-81cc-54984621643d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:43.693034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8118fd15-9616-4667-81cc-54984621643d admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:43.693034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8118fd15-9616-4667-81cc-54984621643d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:43.693034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8118fd15-9616-4667-81cc-54984621643d admin admin] PATH_INFO: `/v3/projects/02e2931fabaf4f15960727403906fd80/users/e4d5b2f941f8496e95f7493c8fb0252a/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:43.695455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8118fd15-9616-4667-81cc-54984621643d admin admin] RBAC: Authorizing `identity:create_grant(project_id=02e2931fabaf4f15960727403906fd80, user_id=e4d5b2f941f8496e95f7493c8fb0252a, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:43.698361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8118fd15-9616-4667-81cc-54984621643d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:43.713267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1316/2635] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:40:43 2022] PUT /identity/v3/projects/02e2931fabaf4f15960727403906fd80/users/e4d5b2f941f8496e95f7493c8fb0252a/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:46.145717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedffcae-7524-49dc-8e36-1b0428402e4d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:46.146809 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedffcae-7524-49dc-8e36-1b0428402e4d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:46.147300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedffcae-7524-49dc-8e36-1b0428402e4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:46.168209 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-aedffcae-7524-49dc-8e36-1b0428402e4d None None] MFA Rules not processed for user `434ae422b68e4145ad8ecac27f450cbc`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:46.170116 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-aedffcae-7524-49dc-8e36-1b0428402e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:46.235851 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1320/2636] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:46 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:50.104548 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a818589-8a92-4892-a33b-3f2f2ce8b3dc None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:50.105059 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a818589-8a92-4892-a33b-3f2f2ce8b3dc None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:50.105169 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a818589-8a92-4892-a33b-3f2f2ce8b3dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:50.161061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-9a818589-8a92-4892-a33b-3f2f2ce8b3dc None None] MFA Rules not processed for user `e4d5b2f941f8496e95f7493c8fb0252a`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:50.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9a818589-8a92-4892-a33b-3f2f2ce8b3dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:50.254402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1317/2637] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:50 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:50.267652 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedffcae-7524-49dc-8e36-1b0428402e4d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:50.269353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-aedffcae-7524-49dc-8e36-1b0428402e4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:50.339583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:50.357656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedffcae-7524-49dc-8e36-1b0428402e4d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:50.365992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94ac029a-6360-4d7b-840b-031f0c5af7da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:50.366668 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac029a-6360-4d7b-840b-031f0c5af7da service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:50.366955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac029a-6360-4d7b-840b-031f0c5af7da service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:50.367235 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94ac029a-6360-4d7b-840b-031f0c5af7da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:50.428286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ac029a-6360-4d7b-840b-031f0c5af7da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:50.430200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94ac029a-6360-4d7b-840b-031f0c5af7da service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:50.442286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1321/2638] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:50 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:40:56.128141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a818589-8a92-4892-a33b-3f2f2ce8b3dc None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:56.129682 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9a818589-8a92-4892-a33b-3f2f2ce8b3dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:56.207578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a818589-8a92-4892-a33b-3f2f2ce8b3dc None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:56.212051 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:56.212958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:56.213239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:56.213513 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] PATH_INFO: `/v3/users/3500cd37f2fc41e9bab2f96289b1a737` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:56.214844 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3500cd37f2fc41e9bab2f96289b1a737)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:56.216878 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:56.403461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3500cd37f2fc41e9bab2f96289b1a737'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:56.404125 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3500cd37f2fc41e9bab2f96289b1a737'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:56.410935 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3500cd37f2fc41e9bab2f96289b1a737'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:56.412569 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:40:56.458121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3500cd37f2fc41e9bab2f96289b1a737'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:56.458858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3500cd37f2fc41e9bab2f96289b1a737'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:56.477640 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1318/2639] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:40:56 2022] DELETE /identity/v3/users/3500cd37f2fc41e9bab2f96289b1a737 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:40:56.482135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe622db-44d7-44f4-8339-2b0290f684be None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:56.482630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe622db-44d7-44f4-8339-2b0290f684be None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:56.482743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe622db-44d7-44f4-8339-2b0290f684be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:56.500338 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9fe622db-44d7-44f4-8339-2b0290f684be None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:40:56.502161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9fe622db-44d7-44f4-8339-2b0290f684be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:56.566722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1322/2640] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:40:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:40:56.576329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:56.577738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:56.641887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9508b3af-bad1-49e8-a4d7-9e8cdee95b37 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:56.645658 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0776f265-92e2-48ff-b46e-b18894f06d93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:56.646313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0776f265-92e2-48ff-b46e-b18894f06d93 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:56.646600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0776f265-92e2-48ff-b46e-b18894f06d93 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:56.646882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0776f265-92e2-48ff-b46e-b18894f06d93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:56.656487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0776f265-92e2-48ff-b46e-b18894f06d93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:56.658599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0776f265-92e2-48ff-b46e-b18894f06d93 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:56.670160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1319/2641] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:40:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:40:57.076832 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe622db-44d7-44f4-8339-2b0290f684be None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:40:57.080233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9fe622db-44d7-44f4-8339-2b0290f684be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:40:57.091814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe622db-44d7-44f4-8339-2b0290f684be None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:40:57.097063 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:40:57.097928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:40:57.098223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:40:57.098544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] PATH_INFO: `/v3/projects/111fee5c9a9f4f2984bcbee87576f16d` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:40:57.099698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] RBAC: Authorizing `identity:delete_project(project_id=111fee5c9a9f4f2984bcbee87576f16d)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:40:57.101653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:40:57.191069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '111fee5c9a9f4f2984bcbee87576f16d', 'request_id': 'req-24678a28-9665-4175-965b-a079f46e5cec'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:57.238324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '111fee5c9a9f4f2984bcbee87576f16d', 'request_id': 'req-24678a28-9665-4175-965b-a079f46e5cec'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:57.245219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '111fee5c9a9f4f2984bcbee87576f16d', 'request_id': 'req-24678a28-9665-4175-965b-a079f46e5cec'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:57.256098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] The token cache is being invalidate because project 111fee5c9a9f4f2984bcbee87576f16d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:40:57.260925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:57.260925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:40:57.260925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1323/2642] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:40:57 2022] DELETE /identity/v3/projects/111fee5c9a9f4f2984bcbee87576f16d => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:41:06.665244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:INSERT=1,keystone:DELETE=3,keystone:SELECT=34 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:41:07.253004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=82,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:41:11.890557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988d55c3-f631-4395-bda2-ea7a742e3d77 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:41:11.891063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988d55c3-f631-4395-bda2-ea7a742e3d77 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:41:11.891179 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-988d55c3-f631-4395-bda2-ea7a742e3d77 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:41:11.909077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-988d55c3-f631-4395-bda2-ea7a742e3d77 None None] MFA Rules not processed for user `bf961149e78d4c0c8acb58f1cec447bd`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:41:11.910271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-988d55c3-f631-4395-bda2-ea7a742e3d77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:41:11.966558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1320/2643] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:41:11 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:41:18.603850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:41:18.605305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:41:18.678947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24678a28-9665-4175-965b-a079f46e5cec admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:41:18.682509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:41:18.683140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:41:18.683423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:41:18.683701 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:41:18.685721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:41:18.753182 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:41:18.754600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:41:18.765416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1324/2644] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:41:18 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:41:21.964479 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=27 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:41:22.630722 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988d55c3-f631-4395-bda2-ea7a742e3d77 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:41:22.632119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-988d55c3-f631-4395-bda2-ea7a742e3d77 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:41:22.698481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-988d55c3-f631-4395-bda2-ea7a742e3d77 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:41:22.702139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f8a591-ee1c-4d0d-9b62-f9327fd5577b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:41:22.702787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f8a591-ee1c-4d0d-9b62-f9327fd5577b service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:41:22.703071 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f8a591-ee1c-4d0d-9b62-f9327fd5577b service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:41:22.703376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f8a591-ee1c-4d0d-9b62-f9327fd5577b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:41:22.705161 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c5f8a591-ee1c-4d0d-9b62-f9327fd5577b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:41:22.780557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f8a591-ee1c-4d0d-9b62-f9327fd5577b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:41:22.782037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5f8a591-ee1c-4d0d-9b62-f9327fd5577b service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:41:22.792009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1321/2645] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:41:22 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:41:28.760619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=63 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:41:29.452386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:41:29.453877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:41:29.464553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39cb35c-d79b-4065-a128-37ee5068ffa4 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:41:29.468154 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056e822b-60eb-4a5e-addd-d661b3f350a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:41:29.468818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056e822b-60eb-4a5e-addd-d661b3f350a4 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:41:29.469113 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056e822b-60eb-4a5e-addd-d661b3f350a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:41:29.469435 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056e822b-60eb-4a5e-addd-d661b3f350a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:41:29.471169 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-056e822b-60eb-4a5e-addd-d661b3f350a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:41:29.480538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056e822b-60eb-4a5e-addd-d661b3f350a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:41:29.481992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056e822b-60eb-4a5e-addd-d661b3f350a4 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:41:29.490459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1325/2646] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:41:29 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:41:31.854845 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f8a591-ee1c-4d0d-9b62-f9327fd5577b service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:41:31.864679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f8a591-ee1c-4d0d-9b62-f9327fd5577b service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:41:31.868245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:41:31.868933 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:41:31.869220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:41:31.869495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:41:31.871218 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:41:31.933526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:41:31.934959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:41:31.944495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1322/2647] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:41:31 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:41:39.486304 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:41:41.940585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=87 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:01.782326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056e822b-60eb-4a5e-addd-d661b3f350a4 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:01.791370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056e822b-60eb-4a5e-addd-d661b3f350a4 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:01.795052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6394a446-9ceb-4fc2-9112-39abfec1f9eb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:01.795689 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6394a446-9ceb-4fc2-9112-39abfec1f9eb service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:01.795975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6394a446-9ceb-4fc2-9112-39abfec1f9eb service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:01.796255 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6394a446-9ceb-4fc2-9112-39abfec1f9eb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:01.798028 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6394a446-9ceb-4fc2-9112-39abfec1f9eb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:01.859829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6394a446-9ceb-4fc2-9112-39abfec1f9eb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:01.861229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6394a446-9ceb-4fc2-9112-39abfec1f9eb service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:01.871287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1326/2648] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:42:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:42:05.388979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:05.390506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:05.404487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11da9983-1a46-4e2e-b639-f4bbff576a04 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:05.413972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:05.413972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:05.413972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:05.413972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] PATH_INFO: `/v3/users/434ae422b68e4145ad8ecac27f450cbc` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:05.414416 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=434ae422b68e4145ad8ecac27f450cbc)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:05.415542 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:05.623803 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '434ae422b68e4145ad8ecac27f450cbc'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:05.624474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '434ae422b68e4145ad8ecac27f450cbc'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:05.631351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '434ae422b68e4145ad8ecac27f450cbc'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:05.695205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '434ae422b68e4145ad8ecac27f450cbc'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:05.695872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-d6dc14fb-7d7e-43d3-8622-66b811f8f2a2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '434ae422b68e4145ad8ecac27f450cbc'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:05.713606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1323/2649] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:42:05 2022] DELETE /identity/v3/users/434ae422b68e4145ad8ecac27f450cbc => generated 0 bytes in 325 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:05.971276 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6394a446-9ceb-4fc2-9112-39abfec1f9eb service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:05.982989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6394a446-9ceb-4fc2-9112-39abfec1f9eb service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:06.041780 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:06.042646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.042931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.043210 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] PATH_INFO: `/v3/projects/eee8e4f43f484abdb588a8f3966d74ad` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.044394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=eee8e4f43f484abdb588a8f3966d74ad)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:06.046377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:06.187343 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eee8e4f43f484abdb588a8f3966d74ad', 'request_id': 'req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:06.255885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eee8e4f43f484abdb588a8f3966d74ad', 'request_id': 'req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:06.262203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eee8e4f43f484abdb588a8f3966d74ad', 'request_id': 'req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:06.272488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] The token cache is being invalidate because project eee8e4f43f484abdb588a8f3966d74ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:42:06.272798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:06.273439 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-ea86e1df-c42b-4dd6-b772-e9bcc352d8fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:06.274981 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1327/2650] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:42:05 2022] DELETE /identity/v3/projects/eee8e4f43f484abdb588a8f3966d74ad => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:06.281011 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b206e8-e316-4018-8b25-998e33ae1495 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.281301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b206e8-e316-4018-8b25-998e33ae1495 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.281597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b206e8-e316-4018-8b25-998e33ae1495 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.299779 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-27b206e8-e316-4018-8b25-998e33ae1495 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:06.305141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-27b206e8-e316-4018-8b25-998e33ae1495 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:06.359272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1324/2651] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:42:06.367929 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009230c5-f2d8-4fa9-b4b6-8330fbfc4a4a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.368224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009230c5-f2d8-4fa9-b4b6-8330fbfc4a4a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.368508 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009230c5-f2d8-4fa9-b4b6-8330fbfc4a4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.387647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-009230c5-f2d8-4fa9-b4b6-8330fbfc4a4a None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:06.388896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-009230c5-f2d8-4fa9-b4b6-8330fbfc4a4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:06.396656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1328/2652] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:06.400654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b206e8-e316-4018-8b25-998e33ae1495 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:06.410650 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b206e8-e316-4018-8b25-998e33ae1495 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:06.416898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d232a8-99e8-4153-a7cf-d1e0242e8545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:06.416898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d232a8-99e8-4153-a7cf-d1e0242e8545 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.416898 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d232a8-99e8-4153-a7cf-d1e0242e8545 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.417224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d232a8-99e8-4153-a7cf-d1e0242e8545 admin admin] PATH_INFO: `/v3/domains` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.418123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d232a8-99e8-4153-a7cf-d1e0242e8545 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80300) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:42:06.419783 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d232a8-99e8-4153-a7cf-d1e0242e8545 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:06.419783 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d232a8-99e8-4153-a7cf-d1e0242e8545 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:06.432091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1325/2653] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:42:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:42:06.436312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009230c5-f2d8-4fa9-b4b6-8330fbfc4a4a None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:06.445890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009230c5-f2d8-4fa9-b4b6-8330fbfc4a4a None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:06.450567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ca7082-04d3-4c72-9d9a-948d497d59a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:06.451225 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ca7082-04d3-4c72-9d9a-948d497d59a0 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.451513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ca7082-04d3-4c72-9d9a-948d497d59a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.451795 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ca7082-04d3-4c72-9d9a-948d497d59a0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.452481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ca7082-04d3-4c72-9d9a-948d497d59a0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:06.454421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ca7082-04d3-4c72-9d9a-948d497d59a0 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:06.468038 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:06.554836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1329/2654] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:42:06 2022] POST /identity/v3/projects => generated 381 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:06.558808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d232a8-99e8-4153-a7cf-d1e0242e8545 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:06.569066 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d232a8-99e8-4153-a7cf-d1e0242e8545 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:06.622851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa5cfdc-74e3-423b-8674-e9e9909c787c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:06.623503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa5cfdc-74e3-423b-8674-e9e9909c787c admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.623786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa5cfdc-74e3-423b-8674-e9e9909c787c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.624064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa5cfdc-74e3-423b-8674-e9e9909c787c admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.624753 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa5cfdc-74e3-423b-8674-e9e9909c787c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:06.626690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aa5cfdc-74e3-423b-8674-e9e9909c787c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:06.633140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:06.712910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1326/2655] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:42:06 2022] POST /identity/v3/users => generated 291 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:06.717352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ca7082-04d3-4c72-9d9a-948d497d59a0 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:06.728531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ca7082-04d3-4c72-9d9a-948d497d59a0 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:06.733153 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5e7b54-7a13-4b55-b392-85bbf1d6b679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:06.733802 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5e7b54-7a13-4b55-b392-85bbf1d6b679 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.734117 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5e7b54-7a13-4b55-b392-85bbf1d6b679 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.734365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5e7b54-7a13-4b55-b392-85bbf1d6b679 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.735085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5e7b54-7a13-4b55-b392-85bbf1d6b679 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:06.736685 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5e7b54-7a13-4b55-b392-85bbf1d6b679 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:06.746835 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1330/2656] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:06 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:06.755541 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa5cfdc-74e3-423b-8674-e9e9909c787c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:06.765700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa5cfdc-74e3-423b-8674-e9e9909c787c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:06.770571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0267531e-4d0e-4dc9-8841-4acf9493c36e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:06.771720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0267531e-4d0e-4dc9-8841-4acf9493c36e admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.772001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0267531e-4d0e-4dc9-8841-4acf9493c36e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.772274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0267531e-4d0e-4dc9-8841-4acf9493c36e admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.773024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0267531e-4d0e-4dc9-8841-4acf9493c36e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:06.774637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0267531e-4d0e-4dc9-8841-4acf9493c36e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:06.784348 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1327/2657] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:06 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:06.788842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5e7b54-7a13-4b55-b392-85bbf1d6b679 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:06.798489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5e7b54-7a13-4b55-b392-85bbf1d6b679 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:06.803081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad953cf6-f01a-434b-a053-b9d97b4d679c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:06.803945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad953cf6-f01a-434b-a053-b9d97b4d679c admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.804251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad953cf6-f01a-434b-a053-b9d97b4d679c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.804529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad953cf6-f01a-434b-a053-b9d97b4d679c admin admin] PATH_INFO: `/v3/projects/d4361565756b4b7d8cf473d23d42670d/users/8928b08c4f0f4b3e9ac7b61efa57ce00/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.820687 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad953cf6-f01a-434b-a053-b9d97b4d679c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d4361565756b4b7d8cf473d23d42670d, user_id=8928b08c4f0f4b3e9ac7b61efa57ce00, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:06.823337 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad953cf6-f01a-434b-a053-b9d97b4d679c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:06.878959 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1331/2658] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:42:06 2022] PUT /identity/v3/projects/d4361565756b4b7d8cf473d23d42670d/users/8928b08c4f0f4b3e9ac7b61efa57ce00/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:06.889035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b300b9f-fe60-4131-82a9-3f05e59dcdc1 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:06.890359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b300b9f-fe60-4131-82a9-3f05e59dcdc1 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:06.890359 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b300b9f-fe60-4131-82a9-3f05e59dcdc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:06.916422 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-7b300b9f-fe60-4131-82a9-3f05e59dcdc1 None None] MFA Rules not processed for user `8928b08c4f0f4b3e9ac7b61efa57ce00`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:06.917810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7b300b9f-fe60-4131-82a9-3f05e59dcdc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:06.981364 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1328/2659] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:06 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:06.985796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad953cf6-f01a-434b-a053-b9d97b4d679c admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:07.046053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad953cf6-f01a-434b-a053-b9d97b4d679c admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:07.050613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ed5ae8-45c4-4ea7-8476-575dd1d825ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:07.051259 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ed5ae8-45c4-4ea7-8476-575dd1d825ff admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:07.051541 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ed5ae8-45c4-4ea7-8476-575dd1d825ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:07.051819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05ed5ae8-45c4-4ea7-8476-575dd1d825ff admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:07.053158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ed5ae8-45c4-4ea7-8476-575dd1d825ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:07.055041 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05ed5ae8-45c4-4ea7-8476-575dd1d825ff admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:07.130891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1332/2660] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:42:06 2022] POST /identity/v3/projects => generated 379 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:07.134848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b300b9f-fe60-4131-82a9-3f05e59dcdc1 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:07.145318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b300b9f-fe60-4131-82a9-3f05e59dcdc1 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:07.201131 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910d2028-2a18-4cb6-9c73-342f395a3500 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:07.201794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910d2028-2a18-4cb6-9c73-342f395a3500 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:07.202081 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910d2028-2a18-4cb6-9c73-342f395a3500 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:07.202361 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910d2028-2a18-4cb6-9c73-342f395a3500 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:07.203060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910d2028-2a18-4cb6-9c73-342f395a3500 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:07.205039 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910d2028-2a18-4cb6-9c73-342f395a3500 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:07.307641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1329/2661] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:42:07 2022] POST /identity/v3/users => generated 296 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:42:07.311511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ed5ae8-45c4-4ea7-8476-575dd1d825ff admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:07.322877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05ed5ae8-45c4-4ea7-8476-575dd1d825ff admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:07.329720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff50a3-f631-4030-b9e2-eda4f48bd1aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:07.334332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff50a3-f631-4030-b9e2-eda4f48bd1aa admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:07.334332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff50a3-f631-4030-b9e2-eda4f48bd1aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:07.334332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eff50a3-f631-4030-b9e2-eda4f48bd1aa admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:07.334332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eff50a3-f631-4030-b9e2-eda4f48bd1aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:07.335109 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eff50a3-f631-4030-b9e2-eda4f48bd1aa admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:07.345973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1333/2662] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:07 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:07.350261 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910d2028-2a18-4cb6-9c73-342f395a3500 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:07.360148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910d2028-2a18-4cb6-9c73-342f395a3500 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:07.364815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc2ae37-c372-4c8f-ae98-666ac4a4c378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:07.365690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc2ae37-c372-4c8f-ae98-666ac4a4c378 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:07.365976 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc2ae37-c372-4c8f-ae98-666ac4a4c378 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:07.366263 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc2ae37-c372-4c8f-ae98-666ac4a4c378 admin admin] PATH_INFO: `/v3/projects/592c52427dcd4b04adc52761d12414e2/users/ca3b810f39c344ab90bf23bdffaf2dfa/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:07.381112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc2ae37-c372-4c8f-ae98-666ac4a4c378 admin admin] RBAC: Authorizing `identity:create_grant(project_id=592c52427dcd4b04adc52761d12414e2, user_id=ca3b810f39c344ab90bf23bdffaf2dfa, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:07.383611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc2ae37-c372-4c8f-ae98-666ac4a4c378 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:07.484740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1330/2663] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:42:07 2022] PUT /identity/v3/projects/592c52427dcd4b04adc52761d12414e2/users/ca3b810f39c344ab90bf23bdffaf2dfa/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:07.489015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff50a3-f631-4030-b9e2-eda4f48bd1aa admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:07.499478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eff50a3-f631-4030-b9e2-eda4f48bd1aa admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:07.556222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1052c27d-6cab-4f20-989d-c3be8f88f214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:07.556911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1052c27d-6cab-4f20-989d-c3be8f88f214 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:07.557192 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1052c27d-6cab-4f20-989d-c3be8f88f214 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:07.557509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1052c27d-6cab-4f20-989d-c3be8f88f214 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:07.558257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1052c27d-6cab-4f20-989d-c3be8f88f214 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:07.559849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1052c27d-6cab-4f20-989d-c3be8f88f214 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:07.570545 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1334/2664] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:07 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:07.574751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc2ae37-c372-4c8f-ae98-666ac4a4c378 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:07.584939 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc2ae37-c372-4c8f-ae98-666ac4a4c378 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:07.589643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2495a5-396d-44ad-a35f-e244ecf3b343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:07.590508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2495a5-396d-44ad-a35f-e244ecf3b343 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:07.594148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2495a5-396d-44ad-a35f-e244ecf3b343 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:07.594148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2495a5-396d-44ad-a35f-e244ecf3b343 admin admin] PATH_INFO: `/v3/projects/592c52427dcd4b04adc52761d12414e2/users/ca3b810f39c344ab90bf23bdffaf2dfa/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:07.594148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2495a5-396d-44ad-a35f-e244ecf3b343 admin admin] RBAC: Authorizing `identity:create_grant(project_id=592c52427dcd4b04adc52761d12414e2, user_id=ca3b810f39c344ab90bf23bdffaf2dfa, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:07.595738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2495a5-396d-44ad-a35f-e244ecf3b343 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:07.605873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1331/2665] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:42:07 2022] PUT /identity/v3/projects/592c52427dcd4b04adc52761d12414e2/users/ca3b810f39c344ab90bf23bdffaf2dfa/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:07.617357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd2b584-7823-4831-b83e-c61d1965bdf0 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:07.617673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd2b584-7823-4831-b83e-c61d1965bdf0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:07.617975 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bd2b584-7823-4831-b83e-c61d1965bdf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:07.643877 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3bd2b584-7823-4831-b83e-c61d1965bdf0 None None] MFA Rules not processed for user `ca3b810f39c344ab90bf23bdffaf2dfa`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:07.645031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3bd2b584-7823-4831-b83e-c61d1965bdf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:07.662265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:07.711389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1335/2666] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:07 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:07.723585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2495a5-396d-44ad-a35f-e244ecf3b343 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:07.727485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-fd2495a5-396d-44ad-a35f-e244ecf3b343 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:07.751555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:07.788483 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2495a5-396d-44ad-a35f-e244ecf3b343 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:07.792168 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe246be-bef9-453a-b9b4-53dca18fa6c8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:07.792841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe246be-bef9-453a-b9b4-53dca18fa6c8 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:07.793130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe246be-bef9-453a-b9b4-53dca18fa6c8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:07.793409 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe246be-bef9-453a-b9b4-53dca18fa6c8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:07.802992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe246be-bef9-453a-b9b4-53dca18fa6c8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:07.804538 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe246be-bef9-453a-b9b4-53dca18fa6c8 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:07.814970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1332/2667] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:42:07 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:42:08.677004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd2b584-7823-4831-b83e-c61d1965bdf0 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:08.686050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bd2b584-7823-4831-b83e-c61d1965bdf0 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:08.690887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310bdec-c5a5-459a-b5c8-215fc200a73c tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca3b810f39c344ab90bf23bdffaf2dfa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '592c52427dcd4b04adc52761d12414e2', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:08.691533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310bdec-c5a5-459a-b5c8-215fc200a73c tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:08.691819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310bdec-c5a5-459a-b5c8-215fc200a73c tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:08.692100 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8310bdec-c5a5-459a-b5c8-215fc200a73c tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:08.692857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310bdec-c5a5-459a-b5c8-215fc200a73c tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:42:08.693716 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310bdec-c5a5-459a-b5c8-215fc200a73c tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:08.695305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8310bdec-c5a5-459a-b5c8-215fc200a73c tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:08.707968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1336/2668] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:42:08.712839 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe246be-bef9-453a-b9b4-53dca18fa6c8 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:08.721929 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe246be-bef9-453a-b9b4-53dca18fa6c8 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:08.726718 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7c2b8c-387f-4379-be31-2855e152e553 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca3b810f39c344ab90bf23bdffaf2dfa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '592c52427dcd4b04adc52761d12414e2', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:08.727377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c2b8c-387f-4379-be31-2855e152e553 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:08.727669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c2b8c-387f-4379-be31-2855e152e553 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:08.727957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7c2b8c-387f-4379-be31-2855e152e553 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:08.728664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7c2b8c-387f-4379-be31-2855e152e553 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:08.730638 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7c2b8c-387f-4379-be31-2855e152e553 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:08.809139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1333/2669] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:42:08 2022] POST /identity/v3/projects => generated 354 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:08.813915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310bdec-c5a5-459a-b5c8-215fc200a73c tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:08.825183 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8310bdec-c5a5-459a-b5c8-215fc200a73c tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:08.881194 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca3b810f39c344ab90bf23bdffaf2dfa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '592c52427dcd4b04adc52761d12414e2', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:08.882179 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:08.882469 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:08.882750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:08.883462 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:42:08.883810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:08.885431 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:08.897527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1337/2670] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:42:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:42:08.932785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7c2b8c-387f-4379-be31-2855e152e553 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:08.944401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7c2b8c-387f-4379-be31-2855e152e553 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:08.949431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ca3b810f39c344ab90bf23bdffaf2dfa', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '592c52427dcd4b04adc52761d12414e2', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:08.950547 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:08.950656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:08.952182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] PATH_INFO: `/v3/projects/57beb2f96f314a82aed7ed7a7db3f987` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:08.952329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Authorizing `identity:delete_project(project_id=57beb2f96f314a82aed7ed7a7db3f987)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:08.954166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:09.062445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '57beb2f96f314a82aed7ed7a7db3f987', 'request_id': 'req-e9015485-5187-4573-908a-b806a3b5ad38'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:09.070907 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '57beb2f96f314a82aed7ed7a7db3f987', 'request_id': 'req-e9015485-5187-4573-908a-b806a3b5ad38'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:09.076302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '57beb2f96f314a82aed7ed7a7db3f987', 'request_id': 'req-e9015485-5187-4573-908a-b806a3b5ad38'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:09.169372 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] The token cache is being invalidate because project 57beb2f96f314a82aed7ed7a7db3f987 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:42:09.169599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:09.170234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e9015485-5187-4573-908a-b806a3b5ad38 tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:09.171857 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1334/2671] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:42:08 2022] DELETE /identity/v3/projects/57beb2f96f314a82aed7ed7a7db3f987 => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:09.836479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:09.838344 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:09.901624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f93ef74-4514-4e70-955a-a2f35148a70b tempest-VolumeQuotaClassesTest-918466707 tempest-VolumeQuotaClassesTest-918466707-project-admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:09.941821 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:09.956386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:09.957329 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:09.957616 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:09.957894 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] PATH_INFO: `/v3/users/8928b08c4f0f4b3e9ac7b61efa57ce00` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:09.959104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8928b08c4f0f4b3e9ac7b61efa57ce00)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:09.961056 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:10.104376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8928b08c4f0f4b3e9ac7b61efa57ce00'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.105197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8928b08c4f0f4b3e9ac7b61efa57ce00'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.105793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8928b08c4f0f4b3e9ac7b61efa57ce00'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.112852 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8928b08c4f0f4b3e9ac7b61efa57ce00'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.192553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8928b08c4f0f4b3e9ac7b61efa57ce00'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.210241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1338/2672] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:42:09 2022] DELETE /identity/v3/users/8928b08c4f0f4b3e9ac7b61efa57ce00 => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:10.214728 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dff2ff4-9204-4b69-b280-fd7a31e9e646 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:10.215227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dff2ff4-9204-4b69-b280-fd7a31e9e646 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:10.215344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dff2ff4-9204-4b69-b280-fd7a31e9e646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:10.232829 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-1dff2ff4-9204-4b69-b280-fd7a31e9e646 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:10.234048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1dff2ff4-9204-4b69-b280-fd7a31e9e646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:10.290585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1335/2673] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:10.300006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:10.301422 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:10.363266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79bb5a48-bb59-4df9-a2c5-7393493f5bf0 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:10.370062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3d836b-2458-412a-b40a-071df40ba6b8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:10.370956 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3d836b-2458-412a-b40a-071df40ba6b8 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:10.371244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3d836b-2458-412a-b40a-071df40ba6b8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:10.371527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce3d836b-2458-412a-b40a-071df40ba6b8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:10.381081 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3d836b-2458-412a-b40a-071df40ba6b8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:10.382526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce3d836b-2458-412a-b40a-071df40ba6b8 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:10.392060 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1339/2674] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:42:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:42:10.772034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dff2ff4-9204-4b69-b280-fd7a31e9e646 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:10.773520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1dff2ff4-9204-4b69-b280-fd7a31e9e646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:10.785489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dff2ff4-9204-4b69-b280-fd7a31e9e646 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:10.790788 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:10.791711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:10.791988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:10.792274 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] PATH_INFO: `/v3/projects/d4361565756b4b7d8cf473d23d42670d` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:10.793504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d4361565756b4b7d8cf473d23d42670d)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:10.795472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:10.820801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:10.884027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd4361565756b4b7d8cf473d23d42670d', 'request_id': 'req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.889947 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd4361565756b4b7d8cf473d23d42670d', 'request_id': 'req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.895221 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd4361565756b4b7d8cf473d23d42670d', 'request_id': 'req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.957901 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] The token cache is being invalidate because project d4361565756b4b7d8cf473d23d42670d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:42:10.958190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.958840 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:10.960401 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1336/2675] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:42:10 2022] DELETE /identity/v3/projects/d4361565756b4b7d8cf473d23d42670d => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:10.964656 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3d836b-2458-412a-b40a-071df40ba6b8 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:10.974969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce3d836b-2458-412a-b40a-071df40ba6b8 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:11.038854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:11.039758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:11.040066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:11.040341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] PATH_INFO: `/v3/users/ca3b810f39c344ab90bf23bdffaf2dfa` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:11.041554 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca3b810f39c344ab90bf23bdffaf2dfa)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:11.043523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:11.048941 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:11.235477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca3b810f39c344ab90bf23bdffaf2dfa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:11.236140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca3b810f39c344ab90bf23bdffaf2dfa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:11.236986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca3b810f39c344ab90bf23bdffaf2dfa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:11.244306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca3b810f39c344ab90bf23bdffaf2dfa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:11.313459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d96bf119-f55b-420d-9ef9-7f000722b0a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca3b810f39c344ab90bf23bdffaf2dfa'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:11.331683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1340/2676] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:42:10 2022] DELETE /identity/v3/users/ca3b810f39c344ab90bf23bdffaf2dfa => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:11.748991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:11.750532 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:11.765220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fb7d13e-3ba4-4fa7-a659-e2145946a4c3 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:11.821934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:11.822782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:11.823065 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:11.823338 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] PATH_INFO: `/v3/projects/592c52427dcd4b04adc52761d12414e2` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:11.824507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] RBAC: Authorizing `identity:delete_project(project_id=592c52427dcd4b04adc52761d12414e2)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:11.826450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:11.924336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '592c52427dcd4b04adc52761d12414e2', 'request_id': 'req-16836c4e-aadd-4515-bebf-645c4327ff77'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:11.930910 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '592c52427dcd4b04adc52761d12414e2', 'request_id': 'req-16836c4e-aadd-4515-bebf-645c4327ff77'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:11.936381 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '592c52427dcd4b04adc52761d12414e2', 'request_id': 'req-16836c4e-aadd-4515-bebf-645c4327ff77'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:12.003531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] The token cache is being invalidate because project 592c52427dcd4b04adc52761d12414e2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:42:12.003847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:12.004468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-16836c4e-aadd-4515-bebf-645c4327ff77 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:12.007337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1337/2677] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:42:11 2022] DELETE /identity/v3/projects/592c52427dcd4b04adc52761d12414e2 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:12.014184 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6780d4a9-76b8-4262-8dff-aca95c0f4f18 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.015173 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6780d4a9-76b8-4262-8dff-aca95c0f4f18 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.015173 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6780d4a9-76b8-4262-8dff-aca95c0f4f18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.035326 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-6780d4a9-76b8-4262-8dff-aca95c0f4f18 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:12.036542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-6780d4a9-76b8-4262-8dff-aca95c0f4f18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:12.096759 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1341/2678] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:12.104919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b34ec-4474-440e-8549-d0d2a4aaa2eb None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.105140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b34ec-4474-440e-8549-d0d2a4aaa2eb None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.105486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f14b34ec-4474-440e-8549-d0d2a4aaa2eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.125024 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-f14b34ec-4474-440e-8549-d0d2a4aaa2eb None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:12.126231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f14b34ec-4474-440e-8549-d0d2a4aaa2eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:12.134875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1338/2679] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:12.138951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6780d4a9-76b8-4262-8dff-aca95c0f4f18 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:12.148389 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6780d4a9-76b8-4262-8dff-aca95c0f4f18 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:12.153289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48991cd7-91c3-40ac-b0b1-e9ff90cb75dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:12.153965 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48991cd7-91c3-40ac-b0b1-e9ff90cb75dd admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.154246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48991cd7-91c3-40ac-b0b1-e9ff90cb75dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.154517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48991cd7-91c3-40ac-b0b1-e9ff90cb75dd admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.155229 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48991cd7-91c3-40ac-b0b1-e9ff90cb75dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:42:12.155531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48991cd7-91c3-40ac-b0b1-e9ff90cb75dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:12.157151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48991cd7-91c3-40ac-b0b1-e9ff90cb75dd admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:12.168266 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1342/2680] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:42:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:42:12.172204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b34ec-4474-440e-8549-d0d2a4aaa2eb None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:12.181854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f14b34ec-4474-440e-8549-d0d2a4aaa2eb None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:12.186397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97504a23-6a91-4567-977f-321aad4cb031 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:12.187033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97504a23-6a91-4567-977f-321aad4cb031 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.187320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97504a23-6a91-4567-977f-321aad4cb031 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.187601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97504a23-6a91-4567-977f-321aad4cb031 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.188574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97504a23-6a91-4567-977f-321aad4cb031 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:12.190585 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97504a23-6a91-4567-977f-321aad4cb031 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:12.315526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1339/2681] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:42:12 2022] POST /identity/v3/projects => generated 361 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:12.319571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48991cd7-91c3-40ac-b0b1-e9ff90cb75dd admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:12.329732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48991cd7-91c3-40ac-b0b1-e9ff90cb75dd admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:12.385112 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9c1969-8725-4b83-9878-69754206a445 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:12.385767 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9c1969-8725-4b83-9878-69754206a445 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.386050 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9c1969-8725-4b83-9878-69754206a445 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.386327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9c1969-8725-4b83-9878-69754206a445 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.387007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9c1969-8725-4b83-9878-69754206a445 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:12.388934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9c1969-8725-4b83-9878-69754206a445 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:12.473256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1343/2682] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:42:12 2022] POST /identity/v3/users => generated 281 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:12.477825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97504a23-6a91-4567-977f-321aad4cb031 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:12.487916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97504a23-6a91-4567-977f-321aad4cb031 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:12.493102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fa34c3-c529-4d7b-a855-6096144a3cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:12.493749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fa34c3-c529-4d7b-a855-6096144a3cbe admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.494032 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fa34c3-c529-4d7b-a855-6096144a3cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.495026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6fa34c3-c529-4d7b-a855-6096144a3cbe admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.495026 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fa34c3-c529-4d7b-a855-6096144a3cbe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:12.496733 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6fa34c3-c529-4d7b-a855-6096144a3cbe admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:12.498731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:12.507609 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1340/2683] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:12.511406 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9c1969-8725-4b83-9878-69754206a445 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:12.516813 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:12.521908 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9c1969-8725-4b83-9878-69754206a445 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:12.526555 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5873e024-ef0e-4273-9854-1dddad6dc0cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:12.527237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5873e024-ef0e-4273-9854-1dddad6dc0cb admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.527520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5873e024-ef0e-4273-9854-1dddad6dc0cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.527800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5873e024-ef0e-4273-9854-1dddad6dc0cb admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.528521 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5873e024-ef0e-4273-9854-1dddad6dc0cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:12.530224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5873e024-ef0e-4273-9854-1dddad6dc0cb admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:12.541200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1344/2684] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:12.545054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fa34c3-c529-4d7b-a855-6096144a3cbe admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:12.556708 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6fa34c3-c529-4d7b-a855-6096144a3cbe admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:12.561517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54d59aa-2d5c-41c5-ad65-8cae5dd93f4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:12.564041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d59aa-2d5c-41c5-ad65-8cae5dd93f4f admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.564041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d59aa-2d5c-41c5-ad65-8cae5dd93f4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.564041 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54d59aa-2d5c-41c5-ad65-8cae5dd93f4f admin admin] PATH_INFO: `/v3/projects/08f5e8f49a7e482da9000365f2ac213c/users/a83fdba5247c41e182b50d77fea3f3ee/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.577608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54d59aa-2d5c-41c5-ad65-8cae5dd93f4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=08f5e8f49a7e482da9000365f2ac213c, user_id=a83fdba5247c41e182b50d77fea3f3ee, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:12.580312 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54d59aa-2d5c-41c5-ad65-8cae5dd93f4f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:12.672695 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1341/2685] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:42:12 2022] PUT /identity/v3/projects/08f5e8f49a7e482da9000365f2ac213c/users/a83fdba5247c41e182b50d77fea3f3ee/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 128 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:12.684512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d521b-8aed-49ef-b599-950140860bac None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.684834 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d521b-8aed-49ef-b599-950140860bac None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.685121 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d521b-8aed-49ef-b599-950140860bac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.713972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-b48d521b-8aed-49ef-b599-950140860bac None None] MFA Rules not processed for user `a83fdba5247c41e182b50d77fea3f3ee`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:12.715172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-b48d521b-8aed-49ef-b599-950140860bac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:12.777966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1345/2686] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:12 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:12.809245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54d59aa-2d5c-41c5-ad65-8cae5dd93f4f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:12.811086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c54d59aa-2d5c-41c5-ad65-8cae5dd93f4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:12.866787 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54d59aa-2d5c-41c5-ad65-8cae5dd93f4f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:12.870328 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924bc73d-426e-40cd-8cf0-97cd2d277ef5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:12.870959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bc73d-426e-40cd-8cf0-97cd2d277ef5 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.871241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bc73d-426e-40cd-8cf0-97cd2d277ef5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.871516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924bc73d-426e-40cd-8cf0-97cd2d277ef5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.879564 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924bc73d-426e-40cd-8cf0-97cd2d277ef5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:12.881692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924bc73d-426e-40cd-8cf0-97cd2d277ef5 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:12.890219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1342/2687] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:42:12 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:42:12.904013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d521b-8aed-49ef-b599-950140860bac None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:12.964612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d521b-8aed-49ef-b599-950140860bac None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:12.969264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:12.970128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:12.970420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:12.970700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] PATH_INFO: `/v3/users/a83fdba5247c41e182b50d77fea3f3ee` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:12.971897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] RBAC: Authorizing `identity:delete_user(user_id=a83fdba5247c41e182b50d77fea3f3ee)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:12.973857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:13.152974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a83fdba5247c41e182b50d77fea3f3ee'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:13.153989 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a83fdba5247c41e182b50d77fea3f3ee'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:13.155193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a83fdba5247c41e182b50d77fea3f3ee'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:13.163316 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a83fdba5247c41e182b50d77fea3f3ee'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:13.227853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a83fdba5247c41e182b50d77fea3f3ee'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:13.242257 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:13.246529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1346/2688] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:42:12 2022] DELETE /identity/v3/users/a83fdba5247c41e182b50d77fea3f3ee => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:13.251018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2691bf-ee69-4c0c-bcfa-8910fae04843 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:13.251314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2691bf-ee69-4c0c-bcfa-8910fae04843 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:13.251620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2691bf-ee69-4c0c-bcfa-8910fae04843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:13.270160 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5b2691bf-ee69-4c0c-bcfa-8910fae04843 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:13.271363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5b2691bf-ee69-4c0c-bcfa-8910fae04843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:13.333172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1343/2689] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:13.342485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:13.343869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:13.404127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-643bbd3f-5c17-42e5-a91c-aa2809f41faa admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:13.407652 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214d64be-ddd7-4aa5-a9e1-23471eb1ca04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:13.408293 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214d64be-ddd7-4aa5-a9e1-23471eb1ca04 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:13.408580 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214d64be-ddd7-4aa5-a9e1-23471eb1ca04 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:13.408893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214d64be-ddd7-4aa5-a9e1-23471eb1ca04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:13.418496 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214d64be-ddd7-4aa5-a9e1-23471eb1ca04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:13.419931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214d64be-ddd7-4aa5-a9e1-23471eb1ca04 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:13.429308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1347/2690] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:42:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:42:13.812560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2691bf-ee69-4c0c-bcfa-8910fae04843 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:13.814352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5b2691bf-ee69-4c0c-bcfa-8910fae04843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:13.827886 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2691bf-ee69-4c0c-bcfa-8910fae04843 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:13.833203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:13.838318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:13.838318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:13.838318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] PATH_INFO: `/v3/projects/08f5e8f49a7e482da9000365f2ac213c` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:13.838318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] RBAC: Authorizing `identity:delete_project(project_id=08f5e8f49a7e482da9000365f2ac213c)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:13.838846 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:13.936432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08f5e8f49a7e482da9000365f2ac213c', 'request_id': 'req-096d5b9f-2c8b-4591-81bb-3f0db6698d17'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:13.942789 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08f5e8f49a7e482da9000365f2ac213c', 'request_id': 'req-096d5b9f-2c8b-4591-81bb-3f0db6698d17'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:13.948970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08f5e8f49a7e482da9000365f2ac213c', 'request_id': 'req-096d5b9f-2c8b-4591-81bb-3f0db6698d17'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:13.994714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:13.998423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] The token cache is being invalidate because project 08f5e8f49a7e482da9000365f2ac213c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:42:13.998715 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:13.999346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-096d5b9f-2c8b-4591-81bb-3f0db6698d17 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:14.001006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1344/2691] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:42:13 2022] DELETE /identity/v3/projects/08f5e8f49a7e482da9000365f2ac213c => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:14.006418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af442b6-8729-4ba8-8f18-50bf26f4b199 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.006711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af442b6-8729-4ba8-8f18-50bf26f4b199 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.006992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af442b6-8729-4ba8-8f18-50bf26f4b199 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.024863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-8af442b6-8729-4ba8-8f18-50bf26f4b199 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:14.033793 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8af442b6-8729-4ba8-8f18-50bf26f4b199 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:14.084513 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1348/2692] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:14.092077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14671570-1c05-4144-9008-f033ec247143 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.092374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14671570-1c05-4144-9008-f033ec247143 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.092659 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14671570-1c05-4144-9008-f033ec247143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.110452 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-14671570-1c05-4144-9008-f033ec247143 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:14.111687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-14671570-1c05-4144-9008-f033ec247143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:14.119603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1345/2693] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:14.123640 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af442b6-8729-4ba8-8f18-50bf26f4b199 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:14.133665 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af442b6-8729-4ba8-8f18-50bf26f4b199 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:14.138191 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec555fc-3888-4700-a764-059708980051 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:14.138870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec555fc-3888-4700-a764-059708980051 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.139158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec555fc-3888-4700-a764-059708980051 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.139442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec555fc-3888-4700-a764-059708980051 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.140162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec555fc-3888-4700-a764-059708980051 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:42:14.140473 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec555fc-3888-4700-a764-059708980051 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:14.142106 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec555fc-3888-4700-a764-059708980051 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:14.153881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1349/2694] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:42:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:42:14.157908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14671570-1c05-4144-9008-f033ec247143 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:14.166970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14671570-1c05-4144-9008-f033ec247143 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:14.171586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd63329-2b13-4b0e-bdeb-105fe9300ac7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:14.172211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd63329-2b13-4b0e-bdeb-105fe9300ac7 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.172492 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd63329-2b13-4b0e-bdeb-105fe9300ac7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.172793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd63329-2b13-4b0e-bdeb-105fe9300ac7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.173496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd63329-2b13-4b0e-bdeb-105fe9300ac7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:14.175393 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd63329-2b13-4b0e-bdeb-105fe9300ac7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:14.251931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1346/2695] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:42:14 2022] POST /identity/v3/projects => generated 367 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:14.256560 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec555fc-3888-4700-a764-059708980051 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:14.266438 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec555fc-3888-4700-a764-059708980051 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:14.322728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff8e8f3-3388-4161-8f38-b7690187ce4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:14.323399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff8e8f3-3388-4161-8f38-b7690187ce4d admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.323681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff8e8f3-3388-4161-8f38-b7690187ce4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.323958 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff8e8f3-3388-4161-8f38-b7690187ce4d admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.324641 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff8e8f3-3388-4161-8f38-b7690187ce4d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:14.326595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff8e8f3-3388-4161-8f38-b7690187ce4d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:14.422471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1350/2696] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:42:14 2022] POST /identity/v3/users => generated 284 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:14.426906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd63329-2b13-4b0e-bdeb-105fe9300ac7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:14.439552 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd63329-2b13-4b0e-bdeb-105fe9300ac7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:14.444908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2a39d1-f550-47c4-9e7b-0749f516b558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:14.445565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2a39d1-f550-47c4-9e7b-0749f516b558 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.445849 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2a39d1-f550-47c4-9e7b-0749f516b558 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.446126 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2a39d1-f550-47c4-9e7b-0749f516b558 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.446850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2a39d1-f550-47c4-9e7b-0749f516b558 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:14.448432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2a39d1-f550-47c4-9e7b-0749f516b558 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:14.459343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1347/2697] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:14.463420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff8e8f3-3388-4161-8f38-b7690187ce4d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:14.473931 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff8e8f3-3388-4161-8f38-b7690187ce4d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:14.478777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6999349-01ad-493a-b6ef-ef8390cccd14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:14.479430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6999349-01ad-493a-b6ef-ef8390cccd14 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.479712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6999349-01ad-493a-b6ef-ef8390cccd14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.479986 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6999349-01ad-493a-b6ef-ef8390cccd14 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.480702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6999349-01ad-493a-b6ef-ef8390cccd14 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:14.482346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6999349-01ad-493a-b6ef-ef8390cccd14 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:14.497162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2a39d1-f550-47c4-9e7b-0749f516b558 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:14.497584 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1351/2698] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:14.507666 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2a39d1-f550-47c4-9e7b-0749f516b558 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:14.512531 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9690628-56da-4533-87e8-f22317ea3da8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:14.513446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9690628-56da-4533-87e8-f22317ea3da8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.513730 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9690628-56da-4533-87e8-f22317ea3da8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.514009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9690628-56da-4533-87e8-f22317ea3da8 admin admin] PATH_INFO: `/v3/projects/690b5681535448ed97e40d550f83727f/users/bd7595c4768045d989cdc78c3d50cd37/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.529496 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9690628-56da-4533-87e8-f22317ea3da8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=690b5681535448ed97e40d550f83727f, user_id=bd7595c4768045d989cdc78c3d50cd37, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:14.532307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9690628-56da-4533-87e8-f22317ea3da8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:14.606252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1348/2699] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:42:14 2022] PUT /identity/v3/projects/690b5681535448ed97e40d550f83727f/users/bd7595c4768045d989cdc78c3d50cd37/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:14.617279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771c3261-f73e-46e3-a605-df4e81d61f97 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.617572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771c3261-f73e-46e3-a605-df4e81d61f97 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.617854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-771c3261-f73e-46e3-a605-df4e81d61f97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.636447 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:14.645339 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-771c3261-f73e-46e3-a605-df4e81d61f97 None None] MFA Rules not processed for user `bd7595c4768045d989cdc78c3d50cd37`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:14.646551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-771c3261-f73e-46e3-a605-df4e81d61f97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:14.710993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1352/2700] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:14 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:14.722577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9690628-56da-4533-87e8-f22317ea3da8 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:14.724238 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c9690628-56da-4533-87e8-f22317ea3da8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:14.786550 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9690628-56da-4533-87e8-f22317ea3da8 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:14.790746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91680f9-ec00-46a8-baa6-7ac2d8a6eb88 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a83b4eb626343f3aa5b1c9f3fc828fd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:14.791392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91680f9-ec00-46a8-baa6-7ac2d8a6eb88 service glance] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:14.791675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91680f9-ec00-46a8-baa6-7ac2d8a6eb88 service glance] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:14.791951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91680f9-ec00-46a8-baa6-7ac2d8a6eb88 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:14.799984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91680f9-ec00-46a8-baa6-7ac2d8a6eb88 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:14.801459 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91680f9-ec00-46a8-baa6-7ac2d8a6eb88 service glance] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:14.815174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1349/2701] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:42:14 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:42:24.708233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=28 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:24.807451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=53,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:45.829748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771c3261-f73e-46e3-a605-df4e81d61f97 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:45.890619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-771c3261-f73e-46e3-a605-df4e81d61f97 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:45.895307 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:45.896174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:45.896456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:45.897290 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] PATH_INFO: `/v3/users/bd7595c4768045d989cdc78c3d50cd37` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:45.898481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bd7595c4768045d989cdc78c3d50cd37)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:45.900401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:46.099650 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bd7595c4768045d989cdc78c3d50cd37'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:46.100237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bd7595c4768045d989cdc78c3d50cd37'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:46.100896 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bd7595c4768045d989cdc78c3d50cd37'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:46.108312 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bd7595c4768045d989cdc78c3d50cd37'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:46.177838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bd7595c4768045d989cdc78c3d50cd37'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:46.196082 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1353/2702] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:42:45 2022] DELETE /identity/v3/users/bd7595c4768045d989cdc78c3d50cd37 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:46.200559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4630787f-ce4a-42c7-ba68-3f43b30327a2 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:46.201064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4630787f-ce4a-42c7-ba68-3f43b30327a2 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:46.201171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4630787f-ce4a-42c7-ba68-3f43b30327a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:46.218808 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-4630787f-ce4a-42c7-ba68-3f43b30327a2 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:46.220014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4630787f-ce4a-42c7-ba68-3f43b30327a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:46.277347 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1350/2703] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:46.286943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:46.288366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:46.357442 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b48cf64-bca4-4cd9-85cd-f90cd71a5b24 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:46.361265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24a5774-f0b9-4b9d-b45c-77fd7b4d0bbd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:46.361909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24a5774-f0b9-4b9d-b45c-77fd7b4d0bbd service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:46.362198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24a5774-f0b9-4b9d-b45c-77fd7b4d0bbd service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:46.362481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24a5774-f0b9-4b9d-b45c-77fd7b4d0bbd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:46.371417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24a5774-f0b9-4b9d-b45c-77fd7b4d0bbd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:46.372884 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24a5774-f0b9-4b9d-b45c-77fd7b4d0bbd service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:46.382881 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1354/2704] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:42:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:42:46.773906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4630787f-ce4a-42c7-ba68-3f43b30327a2 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:46.775476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-4630787f-ce4a-42c7-ba68-3f43b30327a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:46.789240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4630787f-ce4a-42c7-ba68-3f43b30327a2 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:46.792692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:46.793591 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:46.793881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:46.794250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] PATH_INFO: `/v3/projects/690b5681535448ed97e40d550f83727f` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:46.795476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=690b5681535448ed97e40d550f83727f)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:46.797543 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:46.922219 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '690b5681535448ed97e40d550f83727f', 'request_id': 'req-693a93ec-d48d-4781-aba6-76d8037730cd'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:46.928450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '690b5681535448ed97e40d550f83727f', 'request_id': 'req-693a93ec-d48d-4781-aba6-76d8037730cd'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:46.934356 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '690b5681535448ed97e40d550f83727f', 'request_id': 'req-693a93ec-d48d-4781-aba6-76d8037730cd'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:47.006329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] The token cache is being invalidate because project 690b5681535448ed97e40d550f83727f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:42:47.006626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:47.007336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-693a93ec-d48d-4781-aba6-76d8037730cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:42:47.008756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1351/2705] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:42:46 2022] DELETE /identity/v3/projects/690b5681535448ed97e40d550f83727f => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:47.017582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb98b7-8392-4cc8-bb0f-026389a8d646 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.018079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb98b7-8392-4cc8-bb0f-026389a8d646 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.018189 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbb98b7-8392-4cc8-bb0f-026389a8d646 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.032005 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:47.071585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-0fbb98b7-8392-4cc8-bb0f-026389a8d646 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:47.072915 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0fbb98b7-8392-4cc8-bb0f-026389a8d646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:47.130644 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1355/2706] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:47.139906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef7bc90-a70c-4dbd-b807-94dc59da0717 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.140190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef7bc90-a70c-4dbd-b807-94dc59da0717 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.140465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef7bc90-a70c-4dbd-b807-94dc59da0717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.158652 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-eef7bc90-a70c-4dbd-b807-94dc59da0717 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:47.159959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-eef7bc90-a70c-4dbd-b807-94dc59da0717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:47.167873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1352/2707] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:47.171863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbb98b7-8392-4cc8-bb0f-026389a8d646 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:47.181421 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fbb98b7-8392-4cc8-bb0f-026389a8d646 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:47.186430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a40920-801c-4116-8b0d-1febef7f50ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:47.187075 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a40920-801c-4116-8b0d-1febef7f50ad admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.187361 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a40920-801c-4116-8b0d-1febef7f50ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.187641 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a40920-801c-4116-8b0d-1febef7f50ad admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.188362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a40920-801c-4116-8b0d-1febef7f50ad admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:42:47.188669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a40920-801c-4116-8b0d-1febef7f50ad admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:47.190567 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a40920-801c-4116-8b0d-1febef7f50ad admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:47.202482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1356/2708] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:42:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:42:47.206397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef7bc90-a70c-4dbd-b807-94dc59da0717 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:47.215368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef7bc90-a70c-4dbd-b807-94dc59da0717 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:47.219961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6828b9e-7541-407d-8ab8-cd8606b86f87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:47.220604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6828b9e-7541-407d-8ab8-cd8606b86f87 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.220924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6828b9e-7541-407d-8ab8-cd8606b86f87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.221207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6828b9e-7541-407d-8ab8-cd8606b86f87 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.221908 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6828b9e-7541-407d-8ab8-cd8606b86f87 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:47.224144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6828b9e-7541-407d-8ab8-cd8606b86f87 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:47.311481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1353/2709] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:42:47 2022] POST /identity/v3/projects => generated 365 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:47.316086 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a40920-801c-4116-8b0d-1febef7f50ad admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:47.326503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a40920-801c-4116-8b0d-1febef7f50ad admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:47.384107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a3d9f5-c3d1-478a-9e54-eb2030dca2c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:47.384789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3d9f5-c3d1-478a-9e54-eb2030dca2c2 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.385102 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3d9f5-c3d1-478a-9e54-eb2030dca2c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.385385 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3d9f5-c3d1-478a-9e54-eb2030dca2c2 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.386079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a3d9f5-c3d1-478a-9e54-eb2030dca2c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:47.388068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a3d9f5-c3d1-478a-9e54-eb2030dca2c2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:47.458826 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1357/2710] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:42:47 2022] POST /identity/v3/users => generated 283 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:42:47.463053 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6828b9e-7541-407d-8ab8-cd8606b86f87 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:47.477558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6828b9e-7541-407d-8ab8-cd8606b86f87 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:47.482832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f0bed9-53a9-41b0-bf82-1be33674181d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:47.483527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f0bed9-53a9-41b0-bf82-1be33674181d admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.483811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f0bed9-53a9-41b0-bf82-1be33674181d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.484086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f0bed9-53a9-41b0-bf82-1be33674181d admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.484854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f0bed9-53a9-41b0-bf82-1be33674181d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:47.494736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f0bed9-53a9-41b0-bf82-1be33674181d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:47.496556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1354/2711] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:47 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:47.500531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a3d9f5-c3d1-478a-9e54-eb2030dca2c2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:47.510529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a3d9f5-c3d1-478a-9e54-eb2030dca2c2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:47.515595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f33a2dc2-ab89-4c8f-a3ce-dbca42aa8b17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:47.516248 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33a2dc2-ab89-4c8f-a3ce-dbca42aa8b17 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.516534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33a2dc2-ab89-4c8f-a3ce-dbca42aa8b17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.516848 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f33a2dc2-ab89-4c8f-a3ce-dbca42aa8b17 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.517571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33a2dc2-ab89-4c8f-a3ce-dbca42aa8b17 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:47.519171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f33a2dc2-ab89-4c8f-a3ce-dbca42aa8b17 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:47.529007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1358/2712] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:42:47 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:42:47.533629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f0bed9-53a9-41b0-bf82-1be33674181d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:47.544204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f0bed9-53a9-41b0-bf82-1be33674181d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:47.548856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fe7e5f-63f1-4424-9215-9d54d4e96b4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:47.549725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fe7e5f-63f1-4424-9215-9d54d4e96b4a admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.550012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fe7e5f-63f1-4424-9215-9d54d4e96b4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.550291 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fe7e5f-63f1-4424-9215-9d54d4e96b4a admin admin] PATH_INFO: `/v3/projects/8bb9e74577fa47e6bd8dac23c1dd4a42/users/0c552eaf3c1643ed9d3fc256c3be14ba/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.565351 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fe7e5f-63f1-4424-9215-9d54d4e96b4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bb9e74577fa47e6bd8dac23c1dd4a42, user_id=0c552eaf3c1643ed9d3fc256c3be14ba, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:47.568018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79fe7e5f-63f1-4424-9215-9d54d4e96b4a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:47.663689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1355/2713] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:42:47 2022] PUT /identity/v3/projects/8bb9e74577fa47e6bd8dac23c1dd4a42/users/0c552eaf3c1643ed9d3fc256c3be14ba/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:42:47.673873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e135ff6-7f02-458b-a5eb-59dc422969cd None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.674169 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e135ff6-7f02-458b-a5eb-59dc422969cd None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.674564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e135ff6-7f02-458b-a5eb-59dc422969cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.701023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-0e135ff6-7f02-458b-a5eb-59dc422969cd None None] MFA Rules not processed for user `0c552eaf3c1643ed9d3fc256c3be14ba`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:42:47.702220 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0e135ff6-7f02-458b-a5eb-59dc422969cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:47.742017 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:47.764267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1359/2714] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:42:47 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:42:47.776313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fe7e5f-63f1-4424-9215-9d54d4e96b4a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:42:47.778136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-79fe7e5f-63f1-4424-9215-9d54d4e96b4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:42:47.791406 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:47.834836 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fe7e5f-63f1-4424-9215-9d54d4e96b4a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:42:47.839046 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f05ed7a-8ec1-4141-b135-8997bcfe9c89 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:42:47.839686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f05ed7a-8ec1-4141-b135-8997bcfe9c89 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:42:47.839968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f05ed7a-8ec1-4141-b135-8997bcfe9c89 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:42:47.840245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f05ed7a-8ec1-4141-b135-8997bcfe9c89 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:42:47.848384 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f05ed7a-8ec1-4141-b135-8997bcfe9c89 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:42:47.849843 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f05ed7a-8ec1-4141-b135-8997bcfe9c89 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:42:47.858426 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1356/2715] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:42:47 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:42:57.784136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=8 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:42:57.859645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=21 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:27.960549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e135ff6-7f02-458b-a5eb-59dc422969cd None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:28.019971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e135ff6-7f02-458b-a5eb-59dc422969cd None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:28.024386 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:28.025303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:28.025578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:28.025850 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] PATH_INFO: `/v3/users/0c552eaf3c1643ed9d3fc256c3be14ba` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:28.027026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c552eaf3c1643ed9d3fc256c3be14ba)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:28.028948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:28.188935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c552eaf3c1643ed9d3fc256c3be14ba'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:28.189646 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c552eaf3c1643ed9d3fc256c3be14ba'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:28.190273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c552eaf3c1643ed9d3fc256c3be14ba'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:28.197902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c552eaf3c1643ed9d3fc256c3be14ba'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:28.242872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c552eaf3c1643ed9d3fc256c3be14ba'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:28.260946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1360/2716] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:43:27 2022] DELETE /identity/v3/users/0c552eaf3c1643ed9d3fc256c3be14ba => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:43:28.265723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80228322-dd69-4499-94b3-a7269f93462b None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:28.266223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80228322-dd69-4499-94b3-a7269f93462b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:28.266353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80228322-dd69-4499-94b3-a7269f93462b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:28.284195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-80228322-dd69-4499-94b3-a7269f93462b None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:28.285456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-80228322-dd69-4499-94b3-a7269f93462b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:28.344227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1357/2717] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:43:28.354419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:28.355790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:28.410970 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70080c30-8921-498e-85bf-16c146edeb30 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:28.414568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0746ddaf-1402-425d-a859-d17c3a068397 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:28.415203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0746ddaf-1402-425d-a859-d17c3a068397 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:28.416092 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0746ddaf-1402-425d-a859-d17c3a068397 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:28.416367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0746ddaf-1402-425d-a859-d17c3a068397 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:28.426105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0746ddaf-1402-425d-a859-d17c3a068397 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:28.428629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0746ddaf-1402-425d-a859-d17c3a068397 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:28.440445 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1361/2718] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:43:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:43:28.812429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80228322-dd69-4499-94b3-a7269f93462b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:28.814717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-80228322-dd69-4499-94b3-a7269f93462b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:28.834835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80228322-dd69-4499-94b3-a7269f93462b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:28.840148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:28.841751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:28.842181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:28.842564 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] PATH_INFO: `/v3/projects/8bb9e74577fa47e6bd8dac23c1dd4a42` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:28.843959 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bb9e74577fa47e6bd8dac23c1dd4a42)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:28.845922 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:28.957216 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bb9e74577fa47e6bd8dac23c1dd4a42', 'request_id': 'req-20fa80ac-6854-41d6-bb70-d93952f78044'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:28.963197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bb9e74577fa47e6bd8dac23c1dd4a42', 'request_id': 'req-20fa80ac-6854-41d6-bb70-d93952f78044'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:28.968578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bb9e74577fa47e6bd8dac23c1dd4a42', 'request_id': 'req-20fa80ac-6854-41d6-bb70-d93952f78044'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:29.031478 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] The token cache is being invalidate because project 8bb9e74577fa47e6bd8dac23c1dd4a42 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:43:29.031769 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:29.032394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-20fa80ac-6854-41d6-bb70-d93952f78044 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:29.033969 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1358/2719] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:43:28 2022] DELETE /identity/v3/projects/8bb9e74577fa47e6bd8dac23c1dd4a42 => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:43:29.039141 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54a6fb-ae47-45f0-8e9b-a06cad1311b0 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.039629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54a6fb-ae47-45f0-8e9b-a06cad1311b0 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.039744 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54a6fb-ae47-45f0-8e9b-a06cad1311b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.058378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-2c54a6fb-ae47-45f0-8e9b-a06cad1311b0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:29.059561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2c54a6fb-ae47-45f0-8e9b-a06cad1311b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:29.076265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:29.117543 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1362/2720] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:29.126810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51268a5b-2780-44e3-b1cb-fdbe9ccd8f49 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.127102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51268a5b-2780-44e3-b1cb-fdbe9ccd8f49 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.127381 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51268a5b-2780-44e3-b1cb-fdbe9ccd8f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.145231 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-51268a5b-2780-44e3-b1cb-fdbe9ccd8f49 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:29.149430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-51268a5b-2780-44e3-b1cb-fdbe9ccd8f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:29.154813 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1359/2721] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:29.158932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54a6fb-ae47-45f0-8e9b-a06cad1311b0 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:29.169138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54a6fb-ae47-45f0-8e9b-a06cad1311b0 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:29.174224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac83eec3-e4e8-46ff-a779-c820de242e2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:29.178134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83eec3-e4e8-46ff-a779-c820de242e2e admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.178134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83eec3-e4e8-46ff-a779-c820de242e2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.178134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83eec3-e4e8-46ff-a779-c820de242e2e admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.178134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac83eec3-e4e8-46ff-a779-c820de242e2e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:43:29.178134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac83eec3-e4e8-46ff-a779-c820de242e2e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:29.178134 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac83eec3-e4e8-46ff-a779-c820de242e2e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:29.189611 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1363/2722] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:43:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:43:29.194040 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51268a5b-2780-44e3-b1cb-fdbe9ccd8f49 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:29.203281 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51268a5b-2780-44e3-b1cb-fdbe9ccd8f49 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:29.208141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847528fb-33f9-4430-8766-5d182c013c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:29.208730 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847528fb-33f9-4430-8766-5d182c013c34 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.209058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847528fb-33f9-4430-8766-5d182c013c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.209336 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847528fb-33f9-4430-8766-5d182c013c34 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.210031 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847528fb-33f9-4430-8766-5d182c013c34 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:29.211931 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847528fb-33f9-4430-8766-5d182c013c34 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:29.284812 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1360/2723] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:43:29 2022] POST /identity/v3/projects => generated 375 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:43:29.340014 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac83eec3-e4e8-46ff-a779-c820de242e2e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:29.349972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac83eec3-e4e8-46ff-a779-c820de242e2e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:29.404096 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d2f56-500d-4b34-be2c-a24439bf65c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:29.404752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d2f56-500d-4b34-be2c-a24439bf65c3 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.405072 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d2f56-500d-4b34-be2c-a24439bf65c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.405346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4d2f56-500d-4b34-be2c-a24439bf65c3 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.406019 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4d2f56-500d-4b34-be2c-a24439bf65c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:29.407911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4d2f56-500d-4b34-be2c-a24439bf65c3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:29.558876 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1364/2724] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:43:29 2022] POST /identity/v3/users => generated 288 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:43:29.562781 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847528fb-33f9-4430-8766-5d182c013c34 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:29.574171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847528fb-33f9-4430-8766-5d182c013c34 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:29.578830 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba21358-7f04-42db-8749-c8393dde048b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:29.579474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba21358-7f04-42db-8749-c8393dde048b admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.579758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba21358-7f04-42db-8749-c8393dde048b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.580035 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba21358-7f04-42db-8749-c8393dde048b admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.580746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba21358-7f04-42db-8749-c8393dde048b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:29.582368 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba21358-7f04-42db-8749-c8393dde048b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:29.593608 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1361/2725] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:43:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:43:29.597411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d2f56-500d-4b34-be2c-a24439bf65c3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:29.607427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4d2f56-500d-4b34-be2c-a24439bf65c3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:29.611957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c297ba7c-69b9-497a-86b1-6a2b323d36ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:29.612600 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297ba7c-69b9-497a-86b1-6a2b323d36ac admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.612927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297ba7c-69b9-497a-86b1-6a2b323d36ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.613207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c297ba7c-69b9-497a-86b1-6a2b323d36ac admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.613925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c297ba7c-69b9-497a-86b1-6a2b323d36ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:29.615647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c297ba7c-69b9-497a-86b1-6a2b323d36ac admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:29.625423 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1365/2726] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:43:29 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:43:29.629205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba21358-7f04-42db-8749-c8393dde048b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:29.639244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba21358-7f04-42db-8749-c8393dde048b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:29.643620 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4800cee-efe0-4809-9ebb-3120a254982b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:29.644474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4800cee-efe0-4809-9ebb-3120a254982b admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.644782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4800cee-efe0-4809-9ebb-3120a254982b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.645058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4800cee-efe0-4809-9ebb-3120a254982b admin admin] PATH_INFO: `/v3/projects/ea886c36bcfa4733a55c273baa08ad6a/users/d58a2e01812f4c2f8524b9a93ba45ddb/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.659916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4800cee-efe0-4809-9ebb-3120a254982b admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea886c36bcfa4733a55c273baa08ad6a, user_id=d58a2e01812f4c2f8524b9a93ba45ddb, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:29.662614 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4800cee-efe0-4809-9ebb-3120a254982b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:29.740121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1362/2727] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:43:29 2022] PUT /identity/v3/projects/ea886c36bcfa4733a55c273baa08ad6a/users/d58a2e01812f4c2f8524b9a93ba45ddb/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:43:29.750061 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a955008-2692-4607-8b06-78cf64bb7616 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.750351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a955008-2692-4607-8b06-78cf64bb7616 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.750631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a955008-2692-4607-8b06-78cf64bb7616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.778013 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3a955008-2692-4607-8b06-78cf64bb7616 None None] MFA Rules not processed for user `d58a2e01812f4c2f8524b9a93ba45ddb`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:29.779214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3a955008-2692-4607-8b06-78cf64bb7616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:29.843032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1366/2728] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:29 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:29.853911 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4800cee-efe0-4809-9ebb-3120a254982b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:29.855611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b4800cee-efe0-4809-9ebb-3120a254982b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:29.878709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:29.923529 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4800cee-efe0-4809-9ebb-3120a254982b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:29.927197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6fa8d7-f5ae-484a-8d4e-9071b6e64305 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:29.927914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6fa8d7-f5ae-484a-8d4e-9071b6e64305 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:29.928201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6fa8d7-f5ae-484a-8d4e-9071b6e64305 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:29.928481 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6fa8d7-f5ae-484a-8d4e-9071b6e64305 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:29.937461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6fa8d7-f5ae-484a-8d4e-9071b6e64305 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:29.938900 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6fa8d7-f5ae-484a-8d4e-9071b6e64305 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:29.949180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1363/2729] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:43:29 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:43:39.840076 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:39.974117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=22 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:40.944913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a955008-2692-4607-8b06-78cf64bb7616 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:41.009898 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a955008-2692-4607-8b06-78cf64bb7616 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:41.014428 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:41.015287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:41.015569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:41.015845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] PATH_INFO: `/v3/users/d58a2e01812f4c2f8524b9a93ba45ddb` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:41.017046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] RBAC: Authorizing `identity:delete_user(user_id=d58a2e01812f4c2f8524b9a93ba45ddb)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:41.018999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:41.221642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd58a2e01812f4c2f8524b9a93ba45ddb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:41.222379 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd58a2e01812f4c2f8524b9a93ba45ddb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:41.223128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd58a2e01812f4c2f8524b9a93ba45ddb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:41.230171 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd58a2e01812f4c2f8524b9a93ba45ddb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:41.290333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd58a2e01812f4c2f8524b9a93ba45ddb'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:41.311320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1367/2730] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:43:40 2022] DELETE /identity/v3/users/d58a2e01812f4c2f8524b9a93ba45ddb => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:43:41.315765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5d5a5a-6382-464e-ac62-3b998bc1bb4b None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:41.316294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5d5a5a-6382-464e-ac62-3b998bc1bb4b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:41.316556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5d5a5a-6382-464e-ac62-3b998bc1bb4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:41.334387 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ab5d5a5a-6382-464e-ac62-3b998bc1bb4b None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:41.335574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ab5d5a5a-6382-464e-ac62-3b998bc1bb4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:41.394506 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1364/2731] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:41.404631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:41.406811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:41.463757 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68852b0b-d637-48ea-b045-6bcf68757f4f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:41.467957 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39dbc0b7-10ac-438c-8344-dfd317b5121f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:41.468606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbc0b7-10ac-438c-8344-dfd317b5121f service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:41.468990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbc0b7-10ac-438c-8344-dfd317b5121f service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:41.469275 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39dbc0b7-10ac-438c-8344-dfd317b5121f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:41.484503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dbc0b7-10ac-438c-8344-dfd317b5121f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:41.488015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39dbc0b7-10ac-438c-8344-dfd317b5121f service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:41.497991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1368/2732] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:43:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:43:41.907709 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5d5a5a-6382-464e-ac62-3b998bc1bb4b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:41.909311 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ab5d5a5a-6382-464e-ac62-3b998bc1bb4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:41.922921 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5d5a5a-6382-464e-ac62-3b998bc1bb4b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:41.927549 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:41.928413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:41.928698 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:41.929009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] PATH_INFO: `/v3/projects/ea886c36bcfa4733a55c273baa08ad6a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:41.930205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea886c36bcfa4733a55c273baa08ad6a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:41.932128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:42.031752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea886c36bcfa4733a55c273baa08ad6a', 'request_id': 'req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:42.037723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea886c36bcfa4733a55c273baa08ad6a', 'request_id': 'req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:42.043205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea886c36bcfa4733a55c273baa08ad6a', 'request_id': 'req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:42.117098 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] The token cache is being invalidate because project ea886c36bcfa4733a55c273baa08ad6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:43:42.117414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:42.118077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-b6e278d9-d303-4e5a-a68e-1de3c8d7f672 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:43:42.119657 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1365/2733] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:43:41 2022] DELETE /identity/v3/projects/ea886c36bcfa4733a55c273baa08ad6a => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:43:42.125088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ccdf3d-8282-48ba-814f-68b037875ee1 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.125603 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ccdf3d-8282-48ba-814f-68b037875ee1 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.125712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ccdf3d-8282-48ba-814f-68b037875ee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.143349 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-97ccdf3d-8282-48ba-814f-68b037875ee1 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:42.145052 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-97ccdf3d-8282-48ba-814f-68b037875ee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:42.161240 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:42.203391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1369/2734] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:42.211333 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699bd104-68b3-4258-ab66-9b9bf889e9ab None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.211630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699bd104-68b3-4258-ab66-9b9bf889e9ab None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.211917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699bd104-68b3-4258-ab66-9b9bf889e9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.229806 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-699bd104-68b3-4258-ab66-9b9bf889e9ab None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:42.231002 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-699bd104-68b3-4258-ab66-9b9bf889e9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:42.239634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1366/2735] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:42.244002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ccdf3d-8282-48ba-814f-68b037875ee1 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:42.253948 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ccdf3d-8282-48ba-814f-68b037875ee1 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:42.258394 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355aee91-99f3-459d-a71c-772a3c69d96c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:42.259035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355aee91-99f3-459d-a71c-772a3c69d96c admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.259318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355aee91-99f3-459d-a71c-772a3c69d96c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.259595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355aee91-99f3-459d-a71c-772a3c69d96c admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.260306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355aee91-99f3-459d-a71c-772a3c69d96c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:43:42.260612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355aee91-99f3-459d-a71c-772a3c69d96c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:42.262240 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355aee91-99f3-459d-a71c-772a3c69d96c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:42.273618 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1370/2736] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:43:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:43:42.281700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699bd104-68b3-4258-ab66-9b9bf889e9ab None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:42.290775 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-699bd104-68b3-4258-ab66-9b9bf889e9ab None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:42.295765 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77458cb1-013e-439a-a6a0-2ea7cc8e6929 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:42.296407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77458cb1-013e-439a-a6a0-2ea7cc8e6929 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.296689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77458cb1-013e-439a-a6a0-2ea7cc8e6929 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.297000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77458cb1-013e-439a-a6a0-2ea7cc8e6929 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.297686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77458cb1-013e-439a-a6a0-2ea7cc8e6929 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:42.299627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77458cb1-013e-439a-a6a0-2ea7cc8e6929 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:42.353168 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b5a04-b37f-4008-a8e1-bacc72101a65 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.353464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b5a04-b37f-4008-a8e1-bacc72101a65 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.360348 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b5a04-b37f-4008-a8e1-bacc72101a65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.371669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-f72b5a04-b37f-4008-a8e1-bacc72101a65 None None] MFA Rules not processed for user `bf961149e78d4c0c8acb58f1cec447bd`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:42.373530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f72b5a04-b37f-4008-a8e1-bacc72101a65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:42.420710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1367/2737] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:43:42 2022] POST /identity/v3/projects => generated 383 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:43:42.421758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77458cb1-013e-439a-a6a0-2ea7cc8e6929 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:42.427115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1371/2738] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:42 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jun 10 13:43:42.482510 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:42.487470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77458cb1-013e-439a-a6a0-2ea7cc8e6929 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:42.493480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f261583c-ec72-4ef2-9308-fb7c753f4d05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:42.493480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f261583c-ec72-4ef2-9308-fb7c753f4d05 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.493480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f261583c-ec72-4ef2-9308-fb7c753f4d05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.493480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f261583c-ec72-4ef2-9308-fb7c753f4d05 admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.494124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f261583c-ec72-4ef2-9308-fb7c753f4d05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:42.495924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f261583c-ec72-4ef2-9308-fb7c753f4d05 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:42.596800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1368/2739] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:43:42 2022] POST /identity/v3/users => generated 292 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:43:42.600949 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b5a04-b37f-4008-a8e1-bacc72101a65 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:42.612779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b5a04-b37f-4008-a8e1-bacc72101a65 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:42.617472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efba2fe-9961-48c8-be81-d2869f795aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:42.618135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efba2fe-9961-48c8-be81-d2869f795aa2 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.618416 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efba2fe-9961-48c8-be81-d2869f795aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.618694 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efba2fe-9961-48c8-be81-d2869f795aa2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.619414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efba2fe-9961-48c8-be81-d2869f795aa2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:42.621127 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efba2fe-9961-48c8-be81-d2869f795aa2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:42.631481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1372/2740] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:43:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:43:42.635488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f261583c-ec72-4ef2-9308-fb7c753f4d05 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:42.645644 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f261583c-ec72-4ef2-9308-fb7c753f4d05 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:42.650344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5edb58ac-a8aa-4523-a4e2-29852492a22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:42.651009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edb58ac-a8aa-4523-a4e2-29852492a22d admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.651297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edb58ac-a8aa-4523-a4e2-29852492a22d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.651579 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5edb58ac-a8aa-4523-a4e2-29852492a22d admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.652305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edb58ac-a8aa-4523-a4e2-29852492a22d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:42.654684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5edb58ac-a8aa-4523-a4e2-29852492a22d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:42.668724 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1369/2741] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:43:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:43:42.670523 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efba2fe-9961-48c8-be81-d2869f795aa2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:42.680713 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efba2fe-9961-48c8-be81-d2869f795aa2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:42.685501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec82376-038a-4add-a7b6-86835680bca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:42.686383 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec82376-038a-4add-a7b6-86835680bca3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.686664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec82376-038a-4add-a7b6-86835680bca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.686937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec82376-038a-4add-a7b6-86835680bca3 admin admin] PATH_INFO: `/v3/projects/ea8e34865f774cdf830b60908416ee9a/users/ad44d340bc1d4cf48694416948333102/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.703302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec82376-038a-4add-a7b6-86835680bca3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ea8e34865f774cdf830b60908416ee9a, user_id=ad44d340bc1d4cf48694416948333102, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:42.705987 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec82376-038a-4add-a7b6-86835680bca3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:42.758340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1373/2742] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:43:42 2022] PUT /identity/v3/projects/ea8e34865f774cdf830b60908416ee9a/users/ad44d340bc1d4cf48694416948333102/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:43:42.763436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff9e44-0e97-47e6-9703-4845b3bc2a33 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.763731 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff9e44-0e97-47e6-9703-4845b3bc2a33 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.764038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fff9e44-0e97-47e6-9703-4845b3bc2a33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.782135 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5fff9e44-0e97-47e6-9703-4845b3bc2a33 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:42.783350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5fff9e44-0e97-47e6-9703-4845b3bc2a33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:42.850017 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1370/2743] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:42.857935 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec82376-038a-4add-a7b6-86835680bca3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:42.859145 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7ec82376-038a-4add-a7b6-86835680bca3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:42.926865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec82376-038a-4add-a7b6-86835680bca3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:42.930465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f52659a-e0ef-486f-9c37-a93de0a7fdb3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:42.931104 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f52659a-e0ef-486f-9c37-a93de0a7fdb3 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:42.931388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f52659a-e0ef-486f-9c37-a93de0a7fdb3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:42.931690 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f52659a-e0ef-486f-9c37-a93de0a7fdb3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:42.940691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f52659a-e0ef-486f-9c37-a93de0a7fdb3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:42.942148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f52659a-e0ef-486f-9c37-a93de0a7fdb3 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:42.952529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1374/2744] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:43:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:43:48.643054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae5944-614d-4bfa-8069-8128e6d3cf90 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:48.643584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae5944-614d-4bfa-8069-8128e6d3cf90 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:48.643725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0ae5944-614d-4bfa-8069-8128e6d3cf90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:48.675185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-b0ae5944-614d-4bfa-8069-8128e6d3cf90 None None] MFA Rules not processed for user `ad44d340bc1d4cf48694416948333102`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:48.675185 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b0ae5944-614d-4bfa-8069-8128e6d3cf90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:48.751665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1371/2745] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:48 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:48.756143 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f52659a-e0ef-486f-9c37-a93de0a7fdb3 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:48.768232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f52659a-e0ef-486f-9c37-a93de0a7fdb3 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:48.773556 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded4764d-56e6-4381-a568-f12fbe675eba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:48.774427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded4764d-56e6-4381-a568-f12fbe675eba admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:48.774776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded4764d-56e6-4381-a568-f12fbe675eba admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:48.775302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ded4764d-56e6-4381-a568-f12fbe675eba admin admin] PATH_INFO: `/v3/projects` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:48.776085 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded4764d-56e6-4381-a568-f12fbe675eba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:48.778110 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ded4764d-56e6-4381-a568-f12fbe675eba admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:48.845227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:48.852683 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1375/2746] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:43:48 2022] POST /identity/v3/projects => generated 385 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:43:48.857334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ae5944-614d-4bfa-8069-8128e6d3cf90 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:48.871271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0ae5944-614d-4bfa-8069-8128e6d3cf90 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:48.972920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc316831-59cc-46b9-95b0-6203f5b7e7be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:48.973612 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc316831-59cc-46b9-95b0-6203f5b7e7be admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:48.973899 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc316831-59cc-46b9-95b0-6203f5b7e7be admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:48.974180 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc316831-59cc-46b9-95b0-6203f5b7e7be admin admin] PATH_INFO: `/v3/users` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:48.975144 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc316831-59cc-46b9-95b0-6203f5b7e7be admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:48.977204 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc316831-59cc-46b9-95b0-6203f5b7e7be admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:48.987646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:49.092276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1372/2747] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:43:48 2022] POST /identity/v3/users => generated 299 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:43:49.093962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded4764d-56e6-4381-a568-f12fbe675eba admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:49.113283 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ded4764d-56e6-4381-a568-f12fbe675eba admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:49.119079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce89a98-7f1f-428b-8da2-f5debe421665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:49.119773 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce89a98-7f1f-428b-8da2-f5debe421665 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:49.120058 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce89a98-7f1f-428b-8da2-f5debe421665 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:49.120335 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce89a98-7f1f-428b-8da2-f5debe421665 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:49.121688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce89a98-7f1f-428b-8da2-f5debe421665 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:49.123979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce89a98-7f1f-428b-8da2-f5debe421665 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:49.140158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1376/2748] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:43:49 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:43:49.141505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc316831-59cc-46b9-95b0-6203f5b7e7be admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:49.165884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc316831-59cc-46b9-95b0-6203f5b7e7be admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:49.171163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f1b33c-c237-477c-93ac-42b4179946e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:49.178258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f1b33c-c237-477c-93ac-42b4179946e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:49.178258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f1b33c-c237-477c-93ac-42b4179946e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:49.178258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f1b33c-c237-477c-93ac-42b4179946e4 admin admin] PATH_INFO: `/v3/projects/0f1e1d85103341cebcced5108971d9ec/users/0876601a47bd493d99bce3de26a3cfcb/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:49.191346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f1b33c-c237-477c-93ac-42b4179946e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f1e1d85103341cebcced5108971d9ec, user_id=0876601a47bd493d99bce3de26a3cfcb, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:49.194739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f1b33c-c237-477c-93ac-42b4179946e4 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:49.305628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1373/2749] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:43:49 2022] PUT /identity/v3/projects/0f1e1d85103341cebcced5108971d9ec/users/0876601a47bd493d99bce3de26a3cfcb/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:43:49.306827 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce89a98-7f1f-428b-8da2-f5debe421665 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:49.333449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce89a98-7f1f-428b-8da2-f5debe421665 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:49.404698 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a37cafd-065f-412a-8977-2f8fe315e0cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:49.405410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a37cafd-065f-412a-8977-2f8fe315e0cd admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:49.405699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a37cafd-065f-412a-8977-2f8fe315e0cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:49.405984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a37cafd-065f-412a-8977-2f8fe315e0cd admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:49.406717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a37cafd-065f-412a-8977-2f8fe315e0cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:49.408367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a37cafd-065f-412a-8977-2f8fe315e0cd admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:49.419249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1377/2750] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:43:49 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:43:49.423649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f1b33c-c237-477c-93ac-42b4179946e4 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:49.435301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f1b33c-c237-477c-93ac-42b4179946e4 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:49.440245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41d3c3e-01f8-41b5-b397-0239ce45dd41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:49.441181 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41d3c3e-01f8-41b5-b397-0239ce45dd41 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:49.441468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41d3c3e-01f8-41b5-b397-0239ce45dd41 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:49.441749 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b41d3c3e-01f8-41b5-b397-0239ce45dd41 admin admin] PATH_INFO: `/v3/projects/0f1e1d85103341cebcced5108971d9ec/users/0876601a47bd493d99bce3de26a3cfcb/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:49.444414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41d3c3e-01f8-41b5-b397-0239ce45dd41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f1e1d85103341cebcced5108971d9ec, user_id=0876601a47bd493d99bce3de26a3cfcb, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:49.447127 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b41d3c3e-01f8-41b5-b397-0239ce45dd41 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:49.459607 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1374/2751] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:43:49 2022] PUT /identity/v3/projects/0f1e1d85103341cebcced5108971d9ec/users/0876601a47bd493d99bce3de26a3cfcb/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:43:54.787912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3154d85d-f65b-4fe5-8d35-93f36c16fa8a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:54.788438 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3154d85d-f65b-4fe5-8d35-93f36c16fa8a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:54.788549 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3154d85d-f65b-4fe5-8d35-93f36c16fa8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:54.825538 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-3154d85d-f65b-4fe5-8d35-93f36c16fa8a None None] MFA Rules not processed for user `0876601a47bd493d99bce3de26a3cfcb`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:54.828400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-3154d85d-f65b-4fe5-8d35-93f36c16fa8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:54.904027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1378/2752] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:43:54 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:54.915696 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41d3c3e-01f8-41b5-b397-0239ce45dd41 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:54.917721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-b41d3c3e-01f8-41b5-b397-0239ce45dd41 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:55.006010 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b41d3c3e-01f8-41b5-b397-0239ce45dd41 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:55.010264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e7b0b-2d75-4b75-adb8-02893ca2e7bd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a83b4eb626343f3aa5b1c9f3fc828fd', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:55.011021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e7b0b-2d75-4b75-adb8-02893ca2e7bd service glance] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:55.011376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e7b0b-2d75-4b75-adb8-02893ca2e7bd service glance] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:55.011865 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c5e7b0b-2d75-4b75-adb8-02893ca2e7bd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:55.085802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e7b0b-2d75-4b75-adb8-02893ca2e7bd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:55.086791 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c5e7b0b-2d75-4b75-adb8-02893ca2e7bd service glance] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:55.097760 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1375/2753] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:43:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:43:55.326622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cf32e3-9a06-4618-afae-9dca6ada2d7a None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:55.327004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cf32e3-9a06-4618-afae-9dca6ada2d7a None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:55.327292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4cf32e3-9a06-4618-afae-9dca6ada2d7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:55.399845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-d4cf32e3-9a06-4618-afae-9dca6ada2d7a None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:43:55.401365 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d4cf32e3-9a06-4618-afae-9dca6ada2d7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:55.445012 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=100 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:55.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1379/2754] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:43:55 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:43:55.649477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e7b0b-2d75-4b75-adb8-02893ca2e7bd service glance] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:43:55.651192 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6c5e7b0b-2d75-4b75-adb8-02893ca2e7bd service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:43:55.766821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:43:55.795815 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c5e7b0b-2d75-4b75-adb8-02893ca2e7bd service glance] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:43:55.802077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de48cb2c-d192-447d-918f-61ac6d7af7fb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:43:55.802740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de48cb2c-d192-447d-918f-61ac6d7af7fb service swift] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:43:55.803023 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de48cb2c-d192-447d-918f-61ac6d7af7fb service swift] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:43:55.803302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de48cb2c-d192-447d-918f-61ac6d7af7fb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:43:55.812433 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de48cb2c-d192-447d-918f-61ac6d7af7fb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:43:55.814043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de48cb2c-d192-447d-918f-61ac6d7af7fb service swift] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:43:55.822777 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1376/2755] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:43:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jun 10 13:44:05.480943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=12 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:44:05.827825 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=9 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:44:06.424300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219616ed-a9c2-415a-92be-471018c2df90 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:44:06.424596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219616ed-a9c2-415a-92be-471018c2df90 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:44:06.427928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219616ed-a9c2-415a-92be-471018c2df90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:44:06.442973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-219616ed-a9c2-415a-92be-471018c2df90 None None] MFA Rules not processed for user `ad44d340bc1d4cf48694416948333102`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:44:06.444505 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-219616ed-a9c2-415a-92be-471018c2df90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:44:06.452014 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1380/2756] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:44:06 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:44:09.880904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0182ae65-b828-4966-be17-4110e3af8156 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:44:09.885679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0182ae65-b828-4966-be17-4110e3af8156 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:44:09.885679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0182ae65-b828-4966-be17-4110e3af8156 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:44:09.900205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-0182ae65-b828-4966-be17-4110e3af8156 None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:44:09.901486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0182ae65-b828-4966-be17-4110e3af8156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:44:09.909508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1377/2757] 10.222.0.25 () {58 vars in 1009 bytes} [Fri Jun 10 13:44:09 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:44:09.922728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219616ed-a9c2-415a-92be-471018c2df90 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:44:09.934947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219616ed-a9c2-415a-92be-471018c2df90 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:44:09.940204 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed54394-6130-4eb7-b47a-60bde0338bac service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:44:09.940968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed54394-6130-4eb7-b47a-60bde0338bac service swift] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:44:09.941324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed54394-6130-4eb7-b47a-60bde0338bac service swift] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:44:09.941678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eed54394-6130-4eb7-b47a-60bde0338bac service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:44:09.950606 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed54394-6130-4eb7-b47a-60bde0338bac service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:44:09.952854 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eed54394-6130-4eb7-b47a-60bde0338bac service swift] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:44:09.961516 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1381/2758] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:44:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 13:44:19.893746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:44:19.957774 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=9 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:44:41.608872 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0182ae65-b828-4966-be17-4110e3af8156 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:44:41.610327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0182ae65-b828-4966-be17-4110e3af8156 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:44:41.652583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed54394-6130-4eb7-b47a-60bde0338bac service swift] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:44:41.654155 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-eed54394-6130-4eb7-b47a-60bde0338bac service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:44:41.747710 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eed54394-6130-4eb7-b47a-60bde0338bac service swift] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:44:41.753400 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:44:41.754067 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:44:41.754356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:44:41.754638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:44:41.756458 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:44:41.769363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:44:41.771489 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:44:41.781396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1382/2759] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:44:41 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:44:41.799463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0182ae65-b828-4966-be17-4110e3af8156 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:44:41.803045 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf213ca-8bd1-4aaf-80a3-bf64656b712d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:44:41.803687 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf213ca-8bd1-4aaf-80a3-bf64656b712d service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:44:41.803972 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf213ca-8bd1-4aaf-80a3-bf64656b712d service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:44:41.804250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf213ca-8bd1-4aaf-80a3-bf64656b712d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:44:41.813376 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf213ca-8bd1-4aaf-80a3-bf64656b712d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:44:41.814802 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf213ca-8bd1-4aaf-80a3-bf64656b712d service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:44:41.824726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1378/2760] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:44:41 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:44:46.738280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:44:46.740755 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:44:46.829022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4da978a-6c2b-44fc-ab2c-afdf1af809bc service placement] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:44:46.835305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba6ee2a-1496-451b-9d6a-15f179a0c403 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:44:46.835962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba6ee2a-1496-451b-9d6a-15f179a0c403 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:44:46.836249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba6ee2a-1496-451b-9d6a-15f179a0c403 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:44:46.836532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba6ee2a-1496-451b-9d6a-15f179a0c403 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:44:46.838354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5ba6ee2a-1496-451b-9d6a-15f179a0c403 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:44:46.854126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba6ee2a-1496-451b-9d6a-15f179a0c403 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:44:46.855740 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba6ee2a-1496-451b-9d6a-15f179a0c403 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:44:46.866559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1383/2761] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:44:46 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:44:51.823320 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=26 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:44:56.861990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=76 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:45:50.018224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf213ca-8bd1-4aaf-80a3-bf64656b712d service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:45:50.019680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-edf213ca-8bd1-4aaf-80a3-bf64656b712d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:45:50.075930 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf213ca-8bd1-4aaf-80a3-bf64656b712d service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:45:50.079490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e194e98-3374-4887-b0c3-97a516acd880 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:45:50.080128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e194e98-3374-4887-b0c3-97a516acd880 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:45:50.080412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e194e98-3374-4887-b0c3-97a516acd880 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:45:50.080692 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e194e98-3374-4887-b0c3-97a516acd880 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:45:50.082469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8e194e98-3374-4887-b0c3-97a516acd880 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:45:50.139413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e194e98-3374-4887-b0c3-97a516acd880 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:45:50.140912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e194e98-3374-4887-b0c3-97a516acd880 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:45:50.150396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1379/2762] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:45:50 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:45:50.258334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec78d25-05d3-457d-9086-ad5455c65fdc None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:45:50.265756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec78d25-05d3-457d-9086-ad5455c65fdc None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:45:50.265756 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec78d25-05d3-457d-9086-ad5455c65fdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:45:50.277342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-9ec78d25-05d3-457d-9086-ad5455c65fdc None None] MFA Rules not processed for user `bf961149e78d4c0c8acb58f1cec447bd`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:45:50.278533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-9ec78d25-05d3-457d-9086-ad5455c65fdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:45:50.285315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1384/2763] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:45:50 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:45:50.635404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e194e98-3374-4887-b0c3-97a516acd880 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:45:50.635404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e194e98-3374-4887-b0c3-97a516acd880 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:45:50.635404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdb828-c220-4aad-967e-d5a830652f80 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:45:50.635404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdb828-c220-4aad-967e-d5a830652f80 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:45:50.635758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdb828-c220-4aad-967e-d5a830652f80 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:45:50.635758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdb828-c220-4aad-967e-d5a830652f80 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:45:50.635758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-08bdb828-c220-4aad-967e-d5a830652f80 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:45:50.635758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bdb828-c220-4aad-967e-d5a830652f80 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:45:50.635758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bdb828-c220-4aad-967e-d5a830652f80 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:45:50.635758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1380/2764] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:45:50 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:45:51.354839 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec78d25-05d3-457d-9086-ad5455c65fdc None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:45:51.367455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec78d25-05d3-457d-9086-ad5455c65fdc None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:45:51.372438 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8826fc54-4de0-44d2-83a6-5534d3424e12 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:45:51.373120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8826fc54-4de0-44d2-83a6-5534d3424e12 service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:45:51.373408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8826fc54-4de0-44d2-83a6-5534d3424e12 service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:45:51.373691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8826fc54-4de0-44d2-83a6-5534d3424e12 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:45:51.375472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-8826fc54-4de0-44d2-83a6-5534d3424e12 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:45:51.387649 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8826fc54-4de0-44d2-83a6-5534d3424e12 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:45:51.389309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8826fc54-4de0-44d2-83a6-5534d3424e12 service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:45:51.400479 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1385/2765] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:45:51 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:46:00.630820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=54 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:46:01.396239 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=9 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:46:06.679834 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdb828-c220-4aad-967e-d5a830652f80 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:06.693491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdb828-c220-4aad-967e-d5a830652f80 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:06.699091 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f6fe94-fbcd-47be-82eb-9eb0805add5b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:06.700734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f6fe94-fbcd-47be-82eb-9eb0805add5b service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:06.701048 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f6fe94-fbcd-47be-82eb-9eb0805add5b service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:06.701324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65f6fe94-fbcd-47be-82eb-9eb0805add5b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:06.703096 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-65f6fe94-fbcd-47be-82eb-9eb0805add5b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:06.715246 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f6fe94-fbcd-47be-82eb-9eb0805add5b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:06.716831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65f6fe94-fbcd-47be-82eb-9eb0805add5b service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:06.727153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1381/2766] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:46:06 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:46:10.191905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a83c0-f9a8-481b-819a-4a9ac1229686 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:10.192320 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a83c0-f9a8-481b-819a-4a9ac1229686 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:10.193148 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e3a83c0-f9a8-481b-819a-4a9ac1229686 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:10.210503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-5e3a83c0-f9a8-481b-819a-4a9ac1229686 None None] MFA Rules not processed for user `eb2c6f06b300473ebafff4503bb24396`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:46:10.211732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5e3a83c0-f9a8-481b-819a-4a9ac1229686 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:10.219018 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1386/2767] 10.222.0.25 () {58 vars in 967 bytes} [Fri Jun 10 13:46:10 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:46:10.258413 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f6fe94-fbcd-47be-82eb-9eb0805add5b service nova] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:10.269764 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65f6fe94-fbcd-47be-82eb-9eb0805add5b service nova] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:10.274989 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121ad81-3f97-4384-a44c-d6ee9e7413ba service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bb18cd41c9a14d358aac14ed62ef9e02', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:10.275570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121ad81-3f97-4384-a44c-d6ee9e7413ba service swift] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:10.275856 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121ad81-3f97-4384-a44c-d6ee9e7413ba service swift] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:10.276137 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f121ad81-3f97-4384-a44c-d6ee9e7413ba service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:10.284001 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f121ad81-3f97-4384-a44c-d6ee9e7413ba service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:10.285611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f121ad81-3f97-4384-a44c-d6ee9e7413ba service swift] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:10.293565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1382/2768] 10.222.0.25 () {60 vars in 1384 bytes} [Fri Jun 10 13:46:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jun 10 13:46:12.650892 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3a83c0-f9a8-481b-819a-4a9ac1229686 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:12.670992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e3a83c0-f9a8-481b-819a-4a9ac1229686 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:12.678438 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256069df-a59b-4593-80f2-4e74ae020239 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:12.679350 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256069df-a59b-4593-80f2-4e74ae020239 service nova] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:12.679774 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256069df-a59b-4593-80f2-4e74ae020239 service nova] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:12.680227 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-256069df-a59b-4593-80f2-4e74ae020239 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:12.682265 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-256069df-a59b-4593-80f2-4e74ae020239 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:12.700546 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256069df-a59b-4593-80f2-4e74ae020239 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:12.702399 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-256069df-a59b-4593-80f2-4e74ae020239 service nova] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:12.730540 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1387/2769] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:46:12 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:46:15.241751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121ad81-3f97-4384-a44c-d6ee9e7413ba service swift] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:15.242408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-f121ad81-3f97-4384-a44c-d6ee9e7413ba service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:15.266487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f121ad81-3f97-4384-a44c-d6ee9e7413ba service swift] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:15.273617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6068e0-3958-49ce-8043-c6c6e3b99cb5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:15.274864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6068e0-3958-49ce-8043-c6c6e3b99cb5 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:15.275315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6068e0-3958-49ce-8043-c6c6e3b99cb5 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:15.275598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f6068e0-3958-49ce-8043-c6c6e3b99cb5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:15.278297 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-7f6068e0-3958-49ce-8043-c6c6e3b99cb5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:15.296773 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6068e0-3958-49ce-8043-c6c6e3b99cb5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:15.298365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f6068e0-3958-49ce-8043-c6c6e3b99cb5 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:15.309120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1383/2770] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:46:15 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:46:18.931008 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256069df-a59b-4593-80f2-4e74ae020239 service nova] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:18.942250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-256069df-a59b-4593-80f2-4e74ae020239 service nova] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:18.947306 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d60010e-a28b-414d-9b4a-d9104e5bc01d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:18.948271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d60010e-a28b-414d-9b4a-d9104e5bc01d service placement] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:18.948617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d60010e-a28b-414d-9b4a-d9104e5bc01d service placement] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:18.948988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d60010e-a28b-414d-9b4a-d9104e5bc01d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:18.950823 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-5d60010e-a28b-414d-9b4a-d9104e5bc01d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:18.963841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d60010e-a28b-414d-9b4a-d9104e5bc01d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:18.965548 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d60010e-a28b-414d-9b4a-d9104e5bc01d service placement] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:18.976195 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1388/2771] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:46:18 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:46:19.066371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6068e0-3958-49ce-8043-c6c6e3b99cb5 service placement] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:19.131077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f6068e0-3958-49ce-8043-c6c6e3b99cb5 service placement] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:19.135924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:19.136820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:19.137102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:19.137380 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] PATH_INFO: `/v3/users/ad44d340bc1d4cf48694416948333102` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:19.139557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad44d340bc1d4cf48694416948333102)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:19.141577 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:19.349278 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad44d340bc1d4cf48694416948333102'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.349924 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad44d340bc1d4cf48694416948333102'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.356667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad44d340bc1d4cf48694416948333102'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.419477 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad44d340bc1d4cf48694416948333102'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.420093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad44d340bc1d4cf48694416948333102'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.438013 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1384/2772] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:46:19 2022] DELETE /identity/v3/users/ad44d340bc1d4cf48694416948333102 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:46:19.679242 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d60010e-a28b-414d-9b4a-d9104e5bc01d service placement] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:19.689798 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d60010e-a28b-414d-9b4a-d9104e5bc01d service placement] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:19.752553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:19.753432 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:19.753715 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:19.753993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] PATH_INFO: `/v3/projects/ea8e34865f774cdf830b60908416ee9a` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:19.755206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ea8e34865f774cdf830b60908416ee9a)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:19.757144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:19.860181 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ea8e34865f774cdf830b60908416ee9a', 'request_id': 'req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.939216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ea8e34865f774cdf830b60908416ee9a', 'request_id': 'req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.946084 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ea8e34865f774cdf830b60908416ee9a', 'request_id': 'req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.957310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] The token cache is being invalidate because project ea8e34865f774cdf830b60908416ee9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:46:19.957605 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.958755 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:19.960297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1389/2773] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:46:19 2022] DELETE /identity/v3/projects/ea8e34865f774cdf830b60908416ee9a => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:46:19.964294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:19.965720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:19.979060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d62b3e3-24d2-4a39-ac78-2da4d82befb8 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:20.016021 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:46:20.034353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:20.035223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:20.035598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:20.035909 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] PATH_INFO: `/v3/users/0876601a47bd493d99bce3de26a3cfcb` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:20.037758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0876601a47bd493d99bce3de26a3cfcb)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:20.039679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:20.274673 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0876601a47bd493d99bce3de26a3cfcb'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.275355 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0876601a47bd493d99bce3de26a3cfcb'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.282253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0876601a47bd493d99bce3de26a3cfcb'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.351505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0876601a47bd493d99bce3de26a3cfcb'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.352143 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0876601a47bd493d99bce3de26a3cfcb'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.369286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1385/2774] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:46:19 2022] DELETE /identity/v3/users/0876601a47bd493d99bce3de26a3cfcb => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:46:20.621720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:20.623207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:20.636478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc39b6fe-db3a-4b3c-ba90-bc1dc120436b admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:20.687913 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:46:20.694317 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:20.695167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:20.695454 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:20.695733 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] PATH_INFO: `/v3/projects/0f1e1d85103341cebcced5108971d9ec` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:20.696940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f1e1d85103341cebcced5108971d9ec)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:20.699107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:20.827434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f1e1d85103341cebcced5108971d9ec', 'request_id': 'req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.899890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f1e1d85103341cebcced5108971d9ec', 'request_id': 'req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.906327 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f1e1d85103341cebcced5108971d9ec', 'request_id': 'req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.916321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] The token cache is being invalidate because project 0f1e1d85103341cebcced5108971d9ec was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:46:20.916608 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.917297 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:46:20.918804 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1390/2775] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:46:20 2022] DELETE /identity/v3/projects/0f1e1d85103341cebcced5108971d9ec => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:46:22.335011 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:22.336403 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:22.391553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114e0c01-73b8-4332-892a-8d78b767b397 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:22.437202 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb0dc83-89c7-4a73-b934-a75dca48728b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:22.437841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb0dc83-89c7-4a73-b934-a75dca48728b service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:22.438124 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb0dc83-89c7-4a73-b934-a75dca48728b service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:22.438399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb0dc83-89c7-4a73-b934-a75dca48728b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:22.440123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1fb0dc83-89c7-4a73-b934-a75dca48728b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:22.469242 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:46:22.507241 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb0dc83-89c7-4a73-b934-a75dca48728b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:22.508674 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb0dc83-89c7-4a73-b934-a75dca48728b service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:22.526475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1386/2776] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:46:22 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:46:30.923049 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=25,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:46:32.558686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=22 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:46:42.950963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:46:42.952401 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:43.008816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf82c5e-1e7f-4594-bbfe-2832d40737f4 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:46:43.012578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c97582-5619-463f-b08f-50fae49d4730 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:46:43.013267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c97582-5619-463f-b08f-50fae49d4730 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:46:43.013559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c97582-5619-463f-b08f-50fae49d4730 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:46:43.013841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99c97582-5619-463f-b08f-50fae49d4730 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:46:43.015601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-99c97582-5619-463f-b08f-50fae49d4730 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:46:43.025800 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c97582-5619-463f-b08f-50fae49d4730 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:46:43.027234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99c97582-5619-463f-b08f-50fae49d4730 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:46:43.035181 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1391/2777] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:46:42 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:46:53.031681 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=26 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:47:52.831108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb0dc83-89c7-4a73-b934-a75dca48728b service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:47:52.840995 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb0dc83-89c7-4a73-b934-a75dca48728b service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:47:52.844693 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:47:52.845498 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:47:52.845786 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:47:52.846063 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:47:52.848117 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:47:52.918463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:47:52.920175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:47:52.931177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1387/2778] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:47:52 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:48:02.926471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=34 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:08.212161 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c97582-5619-463f-b08f-50fae49d4730 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:08.220930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99c97582-5619-463f-b08f-50fae49d4730 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:08.224702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da784d96-3f16-4f4d-9bfe-255d4b4d80ab service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:08.225369 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da784d96-3f16-4f4d-9bfe-255d4b4d80ab service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:08.225659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da784d96-3f16-4f4d-9bfe-255d4b4d80ab service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:08.225944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da784d96-3f16-4f4d-9bfe-255d4b4d80ab service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:08.227706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-da784d96-3f16-4f4d-9bfe-255d4b4d80ab service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:08.311467 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da784d96-3f16-4f4d-9bfe-255d4b4d80ab service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:08.312947 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da784d96-3f16-4f4d-9bfe-255d4b4d80ab service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:08.333529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1392/2779] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:48:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:48:15.187928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:15.189586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:15.205480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5acbf92-e4b4-4169-9e24-18732f929402 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:15.210140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:15.211018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:15.211305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:15.211584 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] PATH_INFO: `/v3/users/bf961149e78d4c0c8acb58f1cec447bd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:15.213006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bf961149e78d4c0c8acb58f1cec447bd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:15.214914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:15.395604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bf961149e78d4c0c8acb58f1cec447bd'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:15.396201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bf961149e78d4c0c8acb58f1cec447bd'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:15.403008 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bf961149e78d4c0c8acb58f1cec447bd'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:15.486817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bf961149e78d4c0c8acb58f1cec447bd'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:15.487451 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bf961149e78d4c0c8acb58f1cec447bd'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:15.505385 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1388/2780] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:48:15 2022] DELETE /identity/v3/users/bf961149e78d4c0c8acb58f1cec447bd => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:15.754796 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da784d96-3f16-4f4d-9bfe-255d4b4d80ab service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:15.764750 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da784d96-3f16-4f4d-9bfe-255d4b4d80ab service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:15.822583 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:15.823471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:15.823779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:15.825412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] PATH_INFO: `/v3/projects/372f8b4d0d5c4046ad91bea6d709a6ef` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:15.825500 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] RBAC: Authorizing `identity:delete_project(project_id=372f8b4d0d5c4046ad91bea6d709a6ef)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:15.827456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:15.941336 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '372f8b4d0d5c4046ad91bea6d709a6ef', 'request_id': 'req-dcf44aef-3029-4f2d-a6ef-afcded139b71'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:15.992872 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '372f8b4d0d5c4046ad91bea6d709a6ef', 'request_id': 'req-dcf44aef-3029-4f2d-a6ef-afcded139b71'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:15.999237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '372f8b4d0d5c4046ad91bea6d709a6ef', 'request_id': 'req-dcf44aef-3029-4f2d-a6ef-afcded139b71'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.003955 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:16.009836 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] The token cache is being invalidate because project 372f8b4d0d5c4046ad91bea6d709a6ef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:48:16.010090 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.010729 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.012272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1393/2781] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:48:15 2022] DELETE /identity/v3/projects/372f8b4d0d5c4046ad91bea6d709a6ef => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:16.016710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:16.018149 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:16.032594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c242d2e-f406-40ab-84f2-d4a285e171a9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:16.089112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:16.090080 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:16.090436 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:16.090810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] PATH_INFO: `/v3/users/e4d5b2f941f8496e95f7493c8fb0252a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:16.092723 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4d5b2f941f8496e95f7493c8fb0252a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:16.094780 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:16.290244 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4d5b2f941f8496e95f7493c8fb0252a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.290853 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4d5b2f941f8496e95f7493c8fb0252a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.297622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4d5b2f941f8496e95f7493c8fb0252a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.384258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4d5b2f941f8496e95f7493c8fb0252a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.384890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4d5b2f941f8496e95f7493c8fb0252a'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.393662 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:16.403141 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1389/2782] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:48:16 2022] DELETE /identity/v3/users/e4d5b2f941f8496e95f7493c8fb0252a => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:16.694526 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:16.695976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:16.710002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf44aef-3029-4f2d-a6ef-afcded139b71 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:16.768596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:16.769484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:16.769768 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:16.770045 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] PATH_INFO: `/v3/projects/02e2931fabaf4f15960727403906fd80` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:16.771244 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] RBAC: Authorizing `identity:delete_project(project_id=02e2931fabaf4f15960727403906fd80)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:16.773198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:16.890799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '02e2931fabaf4f15960727403906fd80', 'request_id': 'req-d5771fa0-961d-48c8-a70b-f795c913d725'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.952745 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '02e2931fabaf4f15960727403906fd80', 'request_id': 'req-d5771fa0-961d-48c8-a70b-f795c913d725'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.959596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '02e2931fabaf4f15960727403906fd80', 'request_id': 'req-d5771fa0-961d-48c8-a70b-f795c913d725'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.970172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] The token cache is being invalidate because project 02e2931fabaf4f15960727403906fd80 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:48:16.970464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.971123 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:16.972670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1394/2783] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:48:16 2022] DELETE /identity/v3/projects/02e2931fabaf4f15960727403906fd80 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:22.644094 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:22.645518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:22.701626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c28dee0c-d1c0-435d-9b0b-b81c879a6e0b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:22.747334 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fd55ba-0976-48aa-9e8c-e909219971e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:22.748503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd55ba-0976-48aa-9e8c-e909219971e3 service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:22.748810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd55ba-0976-48aa-9e8c-e909219971e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:22.749093 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74fd55ba-0976-48aa-9e8c-e909219971e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:22.750850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-74fd55ba-0976-48aa-9e8c-e909219971e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:22.811121 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fd55ba-0976-48aa-9e8c-e909219971e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:22.812556 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74fd55ba-0976-48aa-9e8c-e909219971e3 service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:22.822102 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1390/2784] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:48:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:48:26.659253 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:26.660755 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:26.674342 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5771fa0-961d-48c8-a70b-f795c913d725 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:26.679126 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:26.680001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:26.680286 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:26.680565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] PATH_INFO: `/v3/users/b1d1712ea523498f92f258383863b906` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:26.681811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b1d1712ea523498f92f258383863b906)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:26.683905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:26.887434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b1d1712ea523498f92f258383863b906'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:26.888637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b1d1712ea523498f92f258383863b906'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:26.889325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b1d1712ea523498f92f258383863b906'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:26.896364 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b1d1712ea523498f92f258383863b906'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:26.955579 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-3a3e7222-a73b-4962-a9a1-adef4cadc613 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b1d1712ea523498f92f258383863b906'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:26.969503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:26.974707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1395/2785] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:48:26 2022] DELETE /identity/v3/users/b1d1712ea523498f92f258383863b906 => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:27.206839 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fd55ba-0976-48aa-9e8c-e909219971e3 service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:27.219446 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74fd55ba-0976-48aa-9e8c-e909219971e3 service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:27.284950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:27.291099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:27.291984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:27.292270 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:27.292551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] PATH_INFO: `/v3/projects/28f8244decff453e98e2d372a97a563c` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:27.293763 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=28f8244decff453e98e2d372a97a563c)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:27.295745 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:27.464937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '28f8244decff453e98e2d372a97a563c', 'request_id': 'req-c8deb1df-61cf-4823-946e-ff075f6ad7a2'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:27.471971 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '28f8244decff453e98e2d372a97a563c', 'request_id': 'req-c8deb1df-61cf-4823-946e-ff075f6ad7a2'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:27.477463 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '28f8244decff453e98e2d372a97a563c', 'request_id': 'req-c8deb1df-61cf-4823-946e-ff075f6ad7a2'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:27.572656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] The token cache is being invalidate because project 28f8244decff453e98e2d372a97a563c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:48:27.572987 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:27.573621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c8deb1df-61cf-4823-946e-ff075f6ad7a2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:27.575710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1391/2786] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:48:27 2022] DELETE /identity/v3/projects/28f8244decff453e98e2d372a97a563c => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:27.581867 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee246f6-e90b-478c-9573-f7a7942ce47e None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:27.582157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee246f6-e90b-478c-9573-f7a7942ce47e None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:27.582440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee246f6-e90b-478c-9573-f7a7942ce47e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:27.609415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-7ee246f6-e90b-478c-9573-f7a7942ce47e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:27.610617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7ee246f6-e90b-478c-9573-f7a7942ce47e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:27.668820 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1396/2787] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:27.677028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506f5af9-1893-4ff5-b036-9e83d14de7ba None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:27.677327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506f5af9-1893-4ff5-b036-9e83d14de7ba None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:27.677611 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506f5af9-1893-4ff5-b036-9e83d14de7ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:27.696123 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-506f5af9-1893-4ff5-b036-9e83d14de7ba None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:27.697372 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-506f5af9-1893-4ff5-b036-9e83d14de7ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:27.705992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1392/2788] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:27.709924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee246f6-e90b-478c-9573-f7a7942ce47e None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:27.719539 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee246f6-e90b-478c-9573-f7a7942ce47e None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:27.723960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295927a7-ae8c-4b67-bd98-2002ba37c175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:27.724612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295927a7-ae8c-4b67-bd98-2002ba37c175 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:27.724925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295927a7-ae8c-4b67-bd98-2002ba37c175 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:27.725202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-295927a7-ae8c-4b67-bd98-2002ba37c175 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:27.725917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295927a7-ae8c-4b67-bd98-2002ba37c175 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:48:27.726222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295927a7-ae8c-4b67-bd98-2002ba37c175 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:27.727815 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-295927a7-ae8c-4b67-bd98-2002ba37c175 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:27.739026 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1397/2789] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:48:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jun 10 13:48:27.743000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506f5af9-1893-4ff5-b036-9e83d14de7ba None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:27.752343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506f5af9-1893-4ff5-b036-9e83d14de7ba None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:27.756937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9aaceb5-a5d0-4c7c-8f08-1ae44308fe0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:27.757571 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9aaceb5-a5d0-4c7c-8f08-1ae44308fe0a admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:27.757850 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9aaceb5-a5d0-4c7c-8f08-1ae44308fe0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:27.758122 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9aaceb5-a5d0-4c7c-8f08-1ae44308fe0a admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:27.758814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9aaceb5-a5d0-4c7c-8f08-1ae44308fe0a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:27.760675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9aaceb5-a5d0-4c7c-8f08-1ae44308fe0a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:27.848981 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1393/2790] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:48:27 2022] POST /identity/v3/projects => generated 385 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:48:27.853031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295927a7-ae8c-4b67-bd98-2002ba37c175 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:27.863107 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-295927a7-ae8c-4b67-bd98-2002ba37c175 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:27.918222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f460e92-57fc-44e9-9005-6e3271b2f95f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:27.918876 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f460e92-57fc-44e9-9005-6e3271b2f95f admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:27.919160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f460e92-57fc-44e9-9005-6e3271b2f95f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:27.919434 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f460e92-57fc-44e9-9005-6e3271b2f95f admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:27.920157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f460e92-57fc-44e9-9005-6e3271b2f95f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:27.922097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f460e92-57fc-44e9-9005-6e3271b2f95f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:28.017054 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1398/2791] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:48:27 2022] POST /identity/v3/users => generated 293 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:48:28.020934 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9aaceb5-a5d0-4c7c-8f08-1ae44308fe0a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:28.030904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9aaceb5-a5d0-4c7c-8f08-1ae44308fe0a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:28.035487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759c1a8c-0123-45ee-8ee6-b2a84dae02e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:28.036138 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759c1a8c-0123-45ee-8ee6-b2a84dae02e9 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:28.036420 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759c1a8c-0123-45ee-8ee6-b2a84dae02e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:28.036694 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-759c1a8c-0123-45ee-8ee6-b2a84dae02e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:28.037561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759c1a8c-0123-45ee-8ee6-b2a84dae02e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:28.039265 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-759c1a8c-0123-45ee-8ee6-b2a84dae02e9 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:28.050665 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1394/2792] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:48:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:48:28.054344 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f460e92-57fc-44e9-9005-6e3271b2f95f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:28.065055 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f460e92-57fc-44e9-9005-6e3271b2f95f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:28.069647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4363c028-0527-43ee-89be-e3fd0e5ccf93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:28.070303 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363c028-0527-43ee-89be-e3fd0e5ccf93 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:28.070587 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363c028-0527-43ee-89be-e3fd0e5ccf93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:28.070869 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4363c028-0527-43ee-89be-e3fd0e5ccf93 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:28.071593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363c028-0527-43ee-89be-e3fd0e5ccf93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:28.073233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4363c028-0527-43ee-89be-e3fd0e5ccf93 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:28.083561 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1399/2793] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:48:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:48:28.087507 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759c1a8c-0123-45ee-8ee6-b2a84dae02e9 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:28.097164 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-759c1a8c-0123-45ee-8ee6-b2a84dae02e9 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:28.101621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f324f3b-f173-4e3c-8bd1-d13e9409cfa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:28.102480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f324f3b-f173-4e3c-8bd1-d13e9409cfa8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:28.102761 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f324f3b-f173-4e3c-8bd1-d13e9409cfa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:28.103034 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f324f3b-f173-4e3c-8bd1-d13e9409cfa8 admin admin] PATH_INFO: `/v3/projects/712acea883074881abb993d97efe6fb2/users/389f5193a266449a8181dd2e86b69f1c/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:28.118590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f324f3b-f173-4e3c-8bd1-d13e9409cfa8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=712acea883074881abb993d97efe6fb2, user_id=389f5193a266449a8181dd2e86b69f1c, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:28.121289 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f324f3b-f173-4e3c-8bd1-d13e9409cfa8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:28.216598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1395/2794] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:48:28 2022] PUT /identity/v3/projects/712acea883074881abb993d97efe6fb2/users/389f5193a266449a8181dd2e86b69f1c/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:28.227070 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8439b2d-8c7b-41b5-89ee-db49512e94b3 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:28.227362 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8439b2d-8c7b-41b5-89ee-db49512e94b3 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:28.227645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8439b2d-8c7b-41b5-89ee-db49512e94b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:28.254430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-f8439b2d-8c7b-41b5-89ee-db49512e94b3 None None] MFA Rules not processed for user `389f5193a266449a8181dd2e86b69f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:28.255626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f8439b2d-8c7b-41b5-89ee-db49512e94b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:28.299152 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:28.318615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1400/2795] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:28 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:28.330167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f324f3b-f173-4e3c-8bd1-d13e9409cfa8 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:28.331913 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1f324f3b-f173-4e3c-8bd1-d13e9409cfa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:28.403533 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f324f3b-f173-4e3c-8bd1-d13e9409cfa8 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:28.409570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b840d47-1590-4f9b-99b9-54188fa75a9f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:28.410230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b840d47-1590-4f9b-99b9-54188fa75a9f service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:28.410487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b840d47-1590-4f9b-99b9-54188fa75a9f service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:28.410759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b840d47-1590-4f9b-99b9-54188fa75a9f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:28.418365 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b840d47-1590-4f9b-99b9-54188fa75a9f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:28.420491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b840d47-1590-4f9b-99b9-54188fa75a9f service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:28.429164 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1396/2796] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:48:28 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:48:37.472441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8439b2d-8c7b-41b5-89ee-db49512e94b3 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:37.528953 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8439b2d-8c7b-41b5-89ee-db49512e94b3 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:37.533950 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:37.534811 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:37.535094 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:37.535370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] PATH_INFO: `/v3/users/389f5193a266449a8181dd2e86b69f1c` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:37.536531 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] RBAC: Authorizing `identity:delete_user(user_id=389f5193a266449a8181dd2e86b69f1c)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:37.538472 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:37.755510 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '389f5193a266449a8181dd2e86b69f1c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:37.756103 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '389f5193a266449a8181dd2e86b69f1c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:37.756712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '389f5193a266449a8181dd2e86b69f1c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:37.763570 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '389f5193a266449a8181dd2e86b69f1c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:37.845033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '389f5193a266449a8181dd2e86b69f1c'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:37.862201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1401/2797] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:48:37 2022] DELETE /identity/v3/users/389f5193a266449a8181dd2e86b69f1c => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:37.866925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8086acb4-9408-4185-9647-24ac2f3a061b None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:37.867374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8086acb4-9408-4185-9647-24ac2f3a061b None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:37.867505 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8086acb4-9408-4185-9647-24ac2f3a061b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:37.884559 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-8086acb4-9408-4185-9647-24ac2f3a061b None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:37.885793 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8086acb4-9408-4185-9647-24ac2f3a061b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:37.890158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:37.944079 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1397/2798] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:37.954370 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:37.955843 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:38.011870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9ce1ea8-613a-42aa-8913-643de65f3fae admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:38.015914 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c983cbb0-47cf-49b4-a78f-2a627ecfde4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:38.016547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c983cbb0-47cf-49b4-a78f-2a627ecfde4b service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:38.016861 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c983cbb0-47cf-49b4-a78f-2a627ecfde4b service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:38.017138 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c983cbb0-47cf-49b4-a78f-2a627ecfde4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:38.025961 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c983cbb0-47cf-49b4-a78f-2a627ecfde4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:38.027388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c983cbb0-47cf-49b4-a78f-2a627ecfde4b service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:38.037481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1402/2799] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:48:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:48:38.440555 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8086acb4-9408-4185-9647-24ac2f3a061b None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:38.442044 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8086acb4-9408-4185-9647-24ac2f3a061b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:38.455033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8086acb4-9408-4185-9647-24ac2f3a061b None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:38.459705 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:38.460630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:38.460890 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:38.461175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] PATH_INFO: `/v3/projects/712acea883074881abb993d97efe6fb2` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:38.462516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] RBAC: Authorizing `identity:delete_project(project_id=712acea883074881abb993d97efe6fb2)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:38.464352 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:38.562302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '712acea883074881abb993d97efe6fb2', 'request_id': 'req-80d5c98b-da1e-452c-9155-723acb75b056'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:38.568288 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '712acea883074881abb993d97efe6fb2', 'request_id': 'req-80d5c98b-da1e-452c-9155-723acb75b056'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:38.574133 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '712acea883074881abb993d97efe6fb2', 'request_id': 'req-80d5c98b-da1e-452c-9155-723acb75b056'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:38.655627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] The token cache is being invalidate because project 712acea883074881abb993d97efe6fb2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:48:38.655914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:38.656526 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-80d5c98b-da1e-452c-9155-723acb75b056 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:38.658100 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1398/2800] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:48:38 2022] DELETE /identity/v3/projects/712acea883074881abb993d97efe6fb2 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:38.663187 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33d36e-0ecd-4e85-9255-f3f214463817 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:38.665456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33d36e-0ecd-4e85-9255-f3f214463817 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:38.665456 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab33d36e-0ecd-4e85-9255-f3f214463817 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:38.681697 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ab33d36e-0ecd-4e85-9255-f3f214463817 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:38.682888 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ab33d36e-0ecd-4e85-9255-f3f214463817 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:38.687751 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:38.741615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1403/2801] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:38.750033 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf0246-abd1-489a-bc59-ac9b232b2024 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:38.750321 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf0246-abd1-489a-bc59-ac9b232b2024 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:38.750596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eaf0246-abd1-489a-bc59-ac9b232b2024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:38.768197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-8eaf0246-abd1-489a-bc59-ac9b232b2024 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:38.769414 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8eaf0246-abd1-489a-bc59-ac9b232b2024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:38.777148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1399/2802] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:38.781277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33d36e-0ecd-4e85-9255-f3f214463817 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:38.791395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab33d36e-0ecd-4e85-9255-f3f214463817 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:38.796542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32103e86-5ac6-4723-995b-f2fef255c113 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:38.797237 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32103e86-5ac6-4723-995b-f2fef255c113 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:38.797525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32103e86-5ac6-4723-995b-f2fef255c113 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:38.797808 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32103e86-5ac6-4723-995b-f2fef255c113 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:38.798534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32103e86-5ac6-4723-995b-f2fef255c113 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:48:38.798844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32103e86-5ac6-4723-995b-f2fef255c113 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:38.800449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32103e86-5ac6-4723-995b-f2fef255c113 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:38.812652 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1404/2803] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:48:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:48:38.816623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf0246-abd1-489a-bc59-ac9b232b2024 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:38.825800 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eaf0246-abd1-489a-bc59-ac9b232b2024 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:38.830450 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3faabe-8e74-4bd4-a561-84e1d399a5c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:38.834201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3faabe-8e74-4bd4-a561-84e1d399a5c6 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:38.834201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3faabe-8e74-4bd4-a561-84e1d399a5c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:38.834201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3faabe-8e74-4bd4-a561-84e1d399a5c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:38.834201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3faabe-8e74-4bd4-a561-84e1d399a5c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:38.834201 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3faabe-8e74-4bd4-a561-84e1d399a5c6 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:38.931973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1400/2804] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:48:38 2022] POST /identity/v3/projects => generated 375 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:48:38.936260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32103e86-5ac6-4723-995b-f2fef255c113 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:38.946280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32103e86-5ac6-4723-995b-f2fef255c113 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:39.003788 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7c0b32-72fe-4dc8-8d33-df6acf086e1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:39.004452 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7c0b32-72fe-4dc8-8d33-df6acf086e1e admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:39.004742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7c0b32-72fe-4dc8-8d33-df6acf086e1e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:39.005055 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7c0b32-72fe-4dc8-8d33-df6acf086e1e admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:39.005743 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7c0b32-72fe-4dc8-8d33-df6acf086e1e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:39.007693 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7c0b32-72fe-4dc8-8d33-df6acf086e1e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:39.075992 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1405/2805] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:48:38 2022] POST /identity/v3/users => generated 288 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:48:39.079954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3faabe-8e74-4bd4-a561-84e1d399a5c6 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:39.090128 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3faabe-8e74-4bd4-a561-84e1d399a5c6 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:39.095205 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b98c936-1a4c-4f6f-934a-3bfa0bb8f15b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:39.095855 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b98c936-1a4c-4f6f-934a-3bfa0bb8f15b admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:39.096252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b98c936-1a4c-4f6f-934a-3bfa0bb8f15b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:39.096784 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b98c936-1a4c-4f6f-934a-3bfa0bb8f15b admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:39.097625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b98c936-1a4c-4f6f-934a-3bfa0bb8f15b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:39.099586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b98c936-1a4c-4f6f-934a-3bfa0bb8f15b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:39.110756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1401/2806] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:48:39 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:48:39.114559 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7c0b32-72fe-4dc8-8d33-df6acf086e1e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:39.124277 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7c0b32-72fe-4dc8-8d33-df6acf086e1e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:39.128878 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf16ba40-b436-44a0-ad0a-d56ac37e305b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:39.129551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf16ba40-b436-44a0-ad0a-d56ac37e305b admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:39.129840 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf16ba40-b436-44a0-ad0a-d56ac37e305b admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:39.130115 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf16ba40-b436-44a0-ad0a-d56ac37e305b admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:39.130838 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf16ba40-b436-44a0-ad0a-d56ac37e305b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:39.132430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf16ba40-b436-44a0-ad0a-d56ac37e305b admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:39.142201 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1406/2807] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:48:39 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:48:39.146171 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b98c936-1a4c-4f6f-934a-3bfa0bb8f15b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:39.156381 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b98c936-1a4c-4f6f-934a-3bfa0bb8f15b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:39.161145 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0a98bd-49f4-4cd4-9607-2d5d9815302e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:39.162011 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0a98bd-49f4-4cd4-9607-2d5d9815302e admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:39.162295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0a98bd-49f4-4cd4-9607-2d5d9815302e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:39.162570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0a98bd-49f4-4cd4-9607-2d5d9815302e admin admin] PATH_INFO: `/v3/projects/c8b90db7b2fa44269a94c63ea2512159/users/9af12e2d68674326a7f217252b583a98/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:39.177238 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0a98bd-49f4-4cd4-9607-2d5d9815302e admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8b90db7b2fa44269a94c63ea2512159, user_id=9af12e2d68674326a7f217252b583a98, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:39.179884 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0a98bd-49f4-4cd4-9607-2d5d9815302e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:39.236299 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1402/2808] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:48:39 2022] PUT /identity/v3/projects/c8b90db7b2fa44269a94c63ea2512159/users/9af12e2d68674326a7f217252b583a98/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:39.247534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d92d89a-4c36-4b11-9251-d2dc2dc46a9d None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:39.247951 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d92d89a-4c36-4b11-9251-d2dc2dc46a9d None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:39.248221 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d92d89a-4c36-4b11-9251-d2dc2dc46a9d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:39.274382 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-7d92d89a-4c36-4b11-9251-d2dc2dc46a9d None None] MFA Rules not processed for user `9af12e2d68674326a7f217252b583a98`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:39.275581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7d92d89a-4c36-4b11-9251-d2dc2dc46a9d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:39.341544 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1407/2809] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:39 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:39.352175 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0a98bd-49f4-4cd4-9607-2d5d9815302e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:39.354508 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5f0a98bd-49f4-4cd4-9607-2d5d9815302e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:39.385363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:39.425487 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0a98bd-49f4-4cd4-9607-2d5d9815302e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:39.430948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b166f9f5-28f7-4333-bbe9-67ae20ab05c3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e9a0496199134369a0ef740c099eed59', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'ResellerAdmin', 'member', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:39.431637 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166f9f5-28f7-4333-bbe9-67ae20ab05c3 service nova] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:39.431920 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166f9f5-28f7-4333-bbe9-67ae20ab05c3 service nova] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:39.432337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166f9f5-28f7-4333-bbe9-67ae20ab05c3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:39.439896 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b166f9f5-28f7-4333-bbe9-67ae20ab05c3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:39.441521 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b166f9f5-28f7-4333-bbe9-67ae20ab05c3 service nova] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:39.450990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1403/2810] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:48:39 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:48:39.655222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d92d89a-4c36-4b11-9251-d2dc2dc46a9d None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:39.660475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:39.716284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d92d89a-4c36-4b11-9251-d2dc2dc46a9d None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:39.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:39.722093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:39.722372 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:39.722645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] PATH_INFO: `/v3/users/9af12e2d68674326a7f217252b583a98` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:39.723889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] RBAC: Authorizing `identity:delete_user(user_id=9af12e2d68674326a7f217252b583a98)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:39.726212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:39.895159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9af12e2d68674326a7f217252b583a98'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:39.895749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9af12e2d68674326a7f217252b583a98'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:39.896361 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9af12e2d68674326a7f217252b583a98'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:39.903156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9af12e2d68674326a7f217252b583a98'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:39.973294 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9af12e2d68674326a7f217252b583a98'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:39.990988 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1408/2811] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:48:39 2022] DELETE /identity/v3/users/9af12e2d68674326a7f217252b583a98 => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:39.995402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1966ff4f-eefd-4e3e-9ab1-94314c9540c0 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:39.996178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1966ff4f-eefd-4e3e-9ab1-94314c9540c0 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:39.996178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1966ff4f-eefd-4e3e-9ab1-94314c9540c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:40.013501 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-1966ff4f-eefd-4e3e-9ab1-94314c9540c0 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:40.014700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1966ff4f-eefd-4e3e-9ab1-94314c9540c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:40.072000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1404/2812] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:40.081704 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:40.083139 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:40.138621 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f52c2c0b-0794-4c26-a561-53bf18836f8f admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:40.142269 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d990cde-fd6d-4f56-9b70-5333d0c6d725 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:40.142912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d990cde-fd6d-4f56-9b70-5333d0c6d725 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:40.143198 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d990cde-fd6d-4f56-9b70-5333d0c6d725 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:40.143478 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d990cde-fd6d-4f56-9b70-5333d0c6d725 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:40.160262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d990cde-fd6d-4f56-9b70-5333d0c6d725 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:40.161680 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d990cde-fd6d-4f56-9b70-5333d0c6d725 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:40.171553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1409/2813] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:48:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:48:40.560828 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1966ff4f-eefd-4e3e-9ab1-94314c9540c0 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:40.562396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-1966ff4f-eefd-4e3e-9ab1-94314c9540c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:40.575163 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1966ff4f-eefd-4e3e-9ab1-94314c9540c0 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:40.581068 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:40.581213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:40.581466 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:40.581857 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] PATH_INFO: `/v3/projects/c8b90db7b2fa44269a94c63ea2512159` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:40.583062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8b90db7b2fa44269a94c63ea2512159)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:40.585294 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:40.696304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8b90db7b2fa44269a94c63ea2512159', 'request_id': 'req-e09dbdb2-093c-4005-814f-a41ba6befaa7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:40.702848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8b90db7b2fa44269a94c63ea2512159', 'request_id': 'req-e09dbdb2-093c-4005-814f-a41ba6befaa7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:40.708350 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8b90db7b2fa44269a94c63ea2512159', 'request_id': 'req-e09dbdb2-093c-4005-814f-a41ba6befaa7'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:40.775447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] The token cache is being invalidate because project c8b90db7b2fa44269a94c63ea2512159 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:48:40.775740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:40.776432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-e09dbdb2-093c-4005-814f-a41ba6befaa7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:48:40.777988 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1405/2814] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:48:40 2022] DELETE /identity/v3/projects/c8b90db7b2fa44269a94c63ea2512159 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:40.783551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b38324-0734-4f62-988c-d62c3006b1bc None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:40.784020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b38324-0734-4f62-988c-d62c3006b1bc None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:40.784108 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b38324-0734-4f62-988c-d62c3006b1bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:40.801575 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-a3b38324-0734-4f62-988c-d62c3006b1bc None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:40.802776 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-a3b38324-0734-4f62-988c-d62c3006b1bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:40.822937 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:40.861352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1410/2815] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:40.869398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0adf62a-b8c7-4cf9-9c83-d28a9a58d42e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:40.869797 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0adf62a-b8c7-4cf9-9c83-d28a9a58d42e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:40.870085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0adf62a-b8c7-4cf9-9c83-d28a9a58d42e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:40.888883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-a0adf62a-b8c7-4cf9-9c83-d28a9a58d42e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:40.890627 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a0adf62a-b8c7-4cf9-9c83-d28a9a58d42e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:40.898553 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1406/2816] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:40.902702 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b38324-0734-4f62-988c-d62c3006b1bc None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:40.912783 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b38324-0734-4f62-988c-d62c3006b1bc None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:40.918002 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b07aa53-00f5-4920-acb2-4c827caf77cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:40.918653 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b07aa53-00f5-4920-acb2-4c827caf77cd admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:40.918940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b07aa53-00f5-4920-acb2-4c827caf77cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:40.919219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b07aa53-00f5-4920-acb2-4c827caf77cd admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:40.919936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b07aa53-00f5-4920-acb2-4c827caf77cd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:48:40.920313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b07aa53-00f5-4920-acb2-4c827caf77cd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:40.921945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b07aa53-00f5-4920-acb2-4c827caf77cd admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:40.933532 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1411/2817] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:48:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:48:40.937726 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0adf62a-b8c7-4cf9-9c83-d28a9a58d42e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:40.946700 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0adf62a-b8c7-4cf9-9c83-d28a9a58d42e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:40.952018 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a27f21f-4e8b-4a5b-a42a-9c21e725fb61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:40.952664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a27f21f-4e8b-4a5b-a42a-9c21e725fb61 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:40.952975 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a27f21f-4e8b-4a5b-a42a-9c21e725fb61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:40.953253 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a27f21f-4e8b-4a5b-a42a-9c21e725fb61 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:40.953937 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a27f21f-4e8b-4a5b-a42a-9c21e725fb61 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:40.955810 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a27f21f-4e8b-4a5b-a42a-9c21e725fb61 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:41.035223 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1407/2818] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:48:40 2022] POST /identity/v3/projects => generated 373 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:48:41.039234 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b07aa53-00f5-4920-acb2-4c827caf77cd admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:41.049593 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b07aa53-00f5-4920-acb2-4c827caf77cd admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:41.105003 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6269c0a-e117-48ec-8972-2a8b540c1fde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:41.105659 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6269c0a-e117-48ec-8972-2a8b540c1fde admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:41.105971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6269c0a-e117-48ec-8972-2a8b540c1fde admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:41.106249 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6269c0a-e117-48ec-8972-2a8b540c1fde admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:41.106919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6269c0a-e117-48ec-8972-2a8b540c1fde admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:41.108917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6269c0a-e117-48ec-8972-2a8b540c1fde admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:41.212308 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1412/2819] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:48:41 2022] POST /identity/v3/users => generated 287 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:48:41.215604 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a27f21f-4e8b-4a5b-a42a-9c21e725fb61 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:41.226227 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a27f21f-4e8b-4a5b-a42a-9c21e725fb61 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:41.230955 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b0e534-5879-42fd-a360-f456303bd57e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:41.231629 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b0e534-5879-42fd-a360-f456303bd57e admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:41.231914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b0e534-5879-42fd-a360-f456303bd57e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:41.232243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b0e534-5879-42fd-a360-f456303bd57e admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:41.232998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b0e534-5879-42fd-a360-f456303bd57e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:41.234592 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b0e534-5879-42fd-a360-f456303bd57e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:41.236561 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:41.245182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1408/2820] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:48:41.249419 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6269c0a-e117-48ec-8972-2a8b540c1fde admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:41.259569 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6269c0a-e117-48ec-8972-2a8b540c1fde admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:41.265223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd2529a-bac6-46c1-b199-44b40449c632 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:41.265902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd2529a-bac6-46c1-b199-44b40449c632 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:41.266187 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd2529a-bac6-46c1-b199-44b40449c632 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:41.273178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd2529a-bac6-46c1-b199-44b40449c632 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:41.273178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd2529a-bac6-46c1-b199-44b40449c632 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:41.273178 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd2529a-bac6-46c1-b199-44b40449c632 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:41.278907 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1413/2821] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:48:41.282996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b0e534-5879-42fd-a360-f456303bd57e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:41.292683 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b0e534-5879-42fd-a360-f456303bd57e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:41.297928 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe3c5fe-a208-4c93-b923-80e7da940d09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:41.298811 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe3c5fe-a208-4c93-b923-80e7da940d09 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:41.299095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe3c5fe-a208-4c93-b923-80e7da940d09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:41.299371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe3c5fe-a208-4c93-b923-80e7da940d09 admin admin] PATH_INFO: `/v3/projects/4c7f7b21bbbb451596a941c8bab58647/users/467af722384641649f99d2b2a49370d1/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:41.313887 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe3c5fe-a208-4c93-b923-80e7da940d09 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c7f7b21bbbb451596a941c8bab58647, user_id=467af722384641649f99d2b2a49370d1, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:41.316517 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe3c5fe-a208-4c93-b923-80e7da940d09 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:41.437083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1409/2822] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:48:41 2022] PUT /identity/v3/projects/4c7f7b21bbbb451596a941c8bab58647/users/467af722384641649f99d2b2a49370d1/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:41.447318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8692ed9-0065-4161-aae7-fbad21371e64 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:41.447612 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8692ed9-0065-4161-aae7-fbad21371e64 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:41.447925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8692ed9-0065-4161-aae7-fbad21371e64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:41.476994 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-d8692ed9-0065-4161-aae7-fbad21371e64 None None] MFA Rules not processed for user `467af722384641649f99d2b2a49370d1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:41.478900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d8692ed9-0065-4161-aae7-fbad21371e64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:41.541816 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1414/2823] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:41 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:41.546848 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe3c5fe-a208-4c93-b923-80e7da940d09 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:41.607337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe3c5fe-a208-4c93-b923-80e7da940d09 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:41.612475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d856d94-8692-4be8-ba30-6544f82797f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:41.613162 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d856d94-8692-4be8-ba30-6544f82797f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:41.613444 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d856d94-8692-4be8-ba30-6544f82797f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:41.613719 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d856d94-8692-4be8-ba30-6544f82797f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:41.614430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d856d94-8692-4be8-ba30-6544f82797f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:41.616316 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d856d94-8692-4be8-ba30-6544f82797f5 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:41.731879 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1410/2824] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:48:41 2022] POST /identity/v3/projects => generated 373 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:48:41.736331 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8692ed9-0065-4161-aae7-fbad21371e64 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:41.746985 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8692ed9-0065-4161-aae7-fbad21371e64 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:41.765302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:41.801909 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f80ba72-12f2-4a6a-97c5-73278eefd828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:41.802576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f80ba72-12f2-4a6a-97c5-73278eefd828 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:41.802889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f80ba72-12f2-4a6a-97c5-73278eefd828 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:41.803174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f80ba72-12f2-4a6a-97c5-73278eefd828 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:41.803857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f80ba72-12f2-4a6a-97c5-73278eefd828 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:41.805819 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f80ba72-12f2-4a6a-97c5-73278eefd828 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:41.896167 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1415/2825] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:48:41 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:48:41.899950 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d856d94-8692-4be8-ba30-6544f82797f5 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:41.910769 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d856d94-8692-4be8-ba30-6544f82797f5 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:41.915460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cd2fff-4cdd-4961-bdff-a6e86cbc6490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:41.916103 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cd2fff-4cdd-4961-bdff-a6e86cbc6490 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:41.916383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cd2fff-4cdd-4961-bdff-a6e86cbc6490 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:41.916655 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16cd2fff-4cdd-4961-bdff-a6e86cbc6490 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:41.917447 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cd2fff-4cdd-4961-bdff-a6e86cbc6490 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:41.919025 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16cd2fff-4cdd-4961-bdff-a6e86cbc6490 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:41.929984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1411/2826] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:48:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:48:41.933972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f80ba72-12f2-4a6a-97c5-73278eefd828 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:41.943865 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f80ba72-12f2-4a6a-97c5-73278eefd828 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:41.948340 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573af595-4b50-4da3-996f-0a5b597b3d9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:41.949232 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573af595-4b50-4da3-996f-0a5b597b3d9e admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:41.949512 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573af595-4b50-4da3-996f-0a5b597b3d9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:41.949784 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573af595-4b50-4da3-996f-0a5b597b3d9e admin admin] PATH_INFO: `/v3/projects/0d030787a34e4ac682c85492749ad1ac/users/6115ef013f7746b89892e20fe4cde3fc/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:41.964321 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573af595-4b50-4da3-996f-0a5b597b3d9e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d030787a34e4ac682c85492749ad1ac, user_id=6115ef013f7746b89892e20fe4cde3fc, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:41.966984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573af595-4b50-4da3-996f-0a5b597b3d9e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:42.030324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1416/2827] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:48:41 2022] PUT /identity/v3/projects/0d030787a34e4ac682c85492749ad1ac/users/6115ef013f7746b89892e20fe4cde3fc/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:42.034186 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cd2fff-4cdd-4961-bdff-a6e86cbc6490 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:42.043913 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16cd2fff-4cdd-4961-bdff-a6e86cbc6490 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:42.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f08639-b97a-4a1c-aba4-cdd8edb9d7de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:42.099174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f08639-b97a-4a1c-aba4-cdd8edb9d7de admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:42.099461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f08639-b97a-4a1c-aba4-cdd8edb9d7de admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:42.099739 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f08639-b97a-4a1c-aba4-cdd8edb9d7de admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:42.100461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f08639-b97a-4a1c-aba4-cdd8edb9d7de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:42.102083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f08639-b97a-4a1c-aba4-cdd8edb9d7de admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:42.112293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1412/2828] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:48:42 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:48:42.116457 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573af595-4b50-4da3-996f-0a5b597b3d9e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:42.126556 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573af595-4b50-4da3-996f-0a5b597b3d9e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:42.131064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd53af6b-9570-48cb-afb8-fcef51408883 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:42.131927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd53af6b-9570-48cb-afb8-fcef51408883 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:42.132207 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd53af6b-9570-48cb-afb8-fcef51408883 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:42.132481 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd53af6b-9570-48cb-afb8-fcef51408883 admin admin] PATH_INFO: `/v3/projects/0d030787a34e4ac682c85492749ad1ac/users/6115ef013f7746b89892e20fe4cde3fc/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:42.134474 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd53af6b-9570-48cb-afb8-fcef51408883 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0d030787a34e4ac682c85492749ad1ac, user_id=6115ef013f7746b89892e20fe4cde3fc, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:42.137140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd53af6b-9570-48cb-afb8-fcef51408883 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:42.147119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1417/2829] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:48:42 2022] PUT /identity/v3/projects/0d030787a34e4ac682c85492749ad1ac/users/6115ef013f7746b89892e20fe4cde3fc/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 31 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:48:42.202471 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dc19ee-3602-41aa-bd28-d0f65624011a None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:42.202759 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dc19ee-3602-41aa-bd28-d0f65624011a None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:42.203043 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dc19ee-3602-41aa-bd28-d0f65624011a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:42.230027 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-d1dc19ee-3602-41aa-bd28-d0f65624011a None None] MFA Rules not processed for user `6115ef013f7746b89892e20fe4cde3fc`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:48:42.231881 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-d1dc19ee-3602-41aa-bd28-d0f65624011a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:42.264664 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:42.300464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1413/2830] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:48:42 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:48:42.311639 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd53af6b-9570-48cb-afb8-fcef51408883 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:42.313347 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-cd53af6b-9570-48cb-afb8-fcef51408883 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:48:42.372617 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd53af6b-9570-48cb-afb8-fcef51408883 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:42.376256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1a348-7b41-49b4-9531-14cd1c733bf1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:42.376932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1a348-7b41-49b4-9531-14cd1c733bf1 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:42.377216 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1a348-7b41-49b4-9531-14cd1c733bf1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:42.377492 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1a348-7b41-49b4-9531-14cd1c733bf1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:42.386245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df1a348-7b41-49b4-9531-14cd1c733bf1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:42.387673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df1a348-7b41-49b4-9531-14cd1c733bf1 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:42.397498 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1418/2831] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:48:42 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:48:42.602656 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dc19ee-3602-41aa-bd28-d0f65624011a None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:48:42.613112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dc19ee-3602-41aa-bd28-d0f65624011a None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:48:42.616544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2b18c00-827b-432e-827c-e72d57901146 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:48:42.617211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b18c00-827b-432e-827c-e72d57901146 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:48:42.617499 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b18c00-827b-432e-827c-e72d57901146 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:48:42.617774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2b18c00-827b-432e-827c-e72d57901146 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:48:42.673871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b18c00-827b-432e-827c-e72d57901146 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:48:42.675293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2b18c00-827b-432e-827c-e72d57901146 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:48:42.683876 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1414/2832] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:48:42 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:48:52.393357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=60,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:48:52.679917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=42 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:30.485911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1a348-7b41-49b4-9531-14cd1c733bf1 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:30.547064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1a348-7b41-49b4-9531-14cd1c733bf1 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:30.552156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:30.553007 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:30.553291 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:30.553568 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] PATH_INFO: `/v3/users/467af722384641649f99d2b2a49370d1` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:30.554749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=467af722384641649f99d2b2a49370d1)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:30.556675 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:30.721936 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '467af722384641649f99d2b2a49370d1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:30.722626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '467af722384641649f99d2b2a49370d1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:30.723246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '467af722384641649f99d2b2a49370d1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:30.730298 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '467af722384641649f99d2b2a49370d1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:30.787966 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '467af722384641649f99d2b2a49370d1'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:30.808088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1419/2833] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:49:30 2022] DELETE /identity/v3/users/467af722384641649f99d2b2a49370d1 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:30.810593 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeba985-e1e4-4b00-9216-1515c187852e None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:30.811047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeba985-e1e4-4b00-9216-1515c187852e None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:30.811153 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceeba985-e1e4-4b00-9216-1515c187852e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:30.829259 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-ceeba985-e1e4-4b00-9216-1515c187852e None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:30.831115 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ceeba985-e1e4-4b00-9216-1515c187852e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:30.887220 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1415/2834] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:30.896810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:30.898193 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:30.963043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02142a56-16ad-4c02-aa5c-49ddf11f42f3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:30.967097 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36bfe2-84f2-45d9-867b-6090f25355cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:30.967738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36bfe2-84f2-45d9-867b-6090f25355cb service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:30.968023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36bfe2-84f2-45d9-867b-6090f25355cb service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:30.968302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36bfe2-84f2-45d9-867b-6090f25355cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:30.977971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed36bfe2-84f2-45d9-867b-6090f25355cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:30.979398 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed36bfe2-84f2-45d9-867b-6090f25355cb service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:30.998015 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:31.002902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1420/2835] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:49:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:49:31.410448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceeba985-e1e4-4b00-9216-1515c187852e None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:31.412837 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-ceeba985-e1e4-4b00-9216-1515c187852e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:31.447210 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceeba985-e1e4-4b00-9216-1515c187852e None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:31.452276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:31.453230 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:31.453511 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:31.453804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] PATH_INFO: `/v3/projects/4c7f7b21bbbb451596a941c8bab58647` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:31.454986 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c7f7b21bbbb451596a941c8bab58647)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:31.457434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:31.624415 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c7f7b21bbbb451596a941c8bab58647', 'request_id': 'req-2a7a4d81-0a10-4032-919b-1dc552c0d319'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:31.630951 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c7f7b21bbbb451596a941c8bab58647', 'request_id': 'req-2a7a4d81-0a10-4032-919b-1dc552c0d319'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:31.636292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c7f7b21bbbb451596a941c8bab58647', 'request_id': 'req-2a7a4d81-0a10-4032-919b-1dc552c0d319'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:31.718667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] The token cache is being invalidate because project 4c7f7b21bbbb451596a941c8bab58647 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:49:31.718957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:31.719597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:31.721182 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1416/2836] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:49:31 2022] DELETE /identity/v3/projects/4c7f7b21bbbb451596a941c8bab58647 => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:31.727684 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36bfe2-84f2-45d9-867b-6090f25355cb service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:31.734829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36bfe2-84f2-45d9-867b-6090f25355cb service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:31.789447 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:31.790313 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:31.790596 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:31.790871 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] PATH_INFO: `/v3/users/6115ef013f7746b89892e20fe4cde3fc` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:31.792053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6115ef013f7746b89892e20fe4cde3fc)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:31.794006 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:31.960655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6115ef013f7746b89892e20fe4cde3fc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:31.961357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6115ef013f7746b89892e20fe4cde3fc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:31.961971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6115ef013f7746b89892e20fe4cde3fc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:31.969024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6115ef013f7746b89892e20fe4cde3fc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:32.025426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1bf56acc-3208-48b9-88c6-3f10b0954e46 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6115ef013f7746b89892e20fe4cde3fc'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:32.042711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1421/2837] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:49:31 2022] DELETE /identity/v3/users/6115ef013f7746b89892e20fe4cde3fc => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:32.413178 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:32.414912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:32.436721 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7a4d81-0a10-4032-919b-1dc552c0d319 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:32.484551 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:32.491622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:32.492475 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:32.492758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:32.493060 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] PATH_INFO: `/v3/projects/0d030787a34e4ac682c85492749ad1ac` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:32.494236 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0d030787a34e4ac682c85492749ad1ac)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:32.496229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:32.581706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0d030787a34e4ac682c85492749ad1ac', 'request_id': 'req-051b68c2-f930-4ce5-9572-300a7d2b3457'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:32.588042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0d030787a34e4ac682c85492749ad1ac', 'request_id': 'req-051b68c2-f930-4ce5-9572-300a7d2b3457'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:32.593622 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0d030787a34e4ac682c85492749ad1ac', 'request_id': 'req-051b68c2-f930-4ce5-9572-300a7d2b3457'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:32.661774 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] The token cache is being invalidate because project 0d030787a34e4ac682c85492749ad1ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:49:32.662064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:32.662689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-051b68c2-f930-4ce5-9572-300a7d2b3457 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:32.664207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1417/2838] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:49:32 2022] DELETE /identity/v3/projects/0d030787a34e4ac682c85492749ad1ac => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:32.669136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34fb3da-9607-43bb-9588-777525565179 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:32.669430 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34fb3da-9607-43bb-9588-777525565179 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:32.669708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e34fb3da-9607-43bb-9588-777525565179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:32.687565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-e34fb3da-9607-43bb-9588-777525565179 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:32.688900 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e34fb3da-9607-43bb-9588-777525565179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:32.748203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1422/2839] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:32.756390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29408923-1a07-4d6f-8c0f-06deff19a7df None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:32.756679 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29408923-1a07-4d6f-8c0f-06deff19a7df None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:32.756980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29408923-1a07-4d6f-8c0f-06deff19a7df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:32.775009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-29408923-1a07-4d6f-8c0f-06deff19a7df None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:32.776196 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-29408923-1a07-4d6f-8c0f-06deff19a7df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:32.783952 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1418/2840] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:32.787930 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34fb3da-9607-43bb-9588-777525565179 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:32.798218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e34fb3da-9607-43bb-9588-777525565179 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:32.812066 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9c1715-2923-4f2b-843d-3aae0059ee30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:32.812724 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9c1715-2923-4f2b-843d-3aae0059ee30 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:32.813030 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9c1715-2923-4f2b-843d-3aae0059ee30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:32.813305 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9c1715-2923-4f2b-843d-3aae0059ee30 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:32.814029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9c1715-2923-4f2b-843d-3aae0059ee30 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:49:32.814334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9c1715-2923-4f2b-843d-3aae0059ee30 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:32.815940 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9c1715-2923-4f2b-843d-3aae0059ee30 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:32.828647 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1423/2841] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:49:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:49:32.832990 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29408923-1a07-4d6f-8c0f-06deff19a7df None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:32.842029 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29408923-1a07-4d6f-8c0f-06deff19a7df None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:32.846667 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757c1d51-c4a9-40ec-93bf-7d9a90d6940c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:32.847315 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757c1d51-c4a9-40ec-93bf-7d9a90d6940c admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:32.847599 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757c1d51-c4a9-40ec-93bf-7d9a90d6940c admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:32.847877 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-757c1d51-c4a9-40ec-93bf-7d9a90d6940c admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:32.848563 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757c1d51-c4a9-40ec-93bf-7d9a90d6940c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:32.850500 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-757c1d51-c4a9-40ec-93bf-7d9a90d6940c admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:32.952208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1419/2842] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:49:32 2022] POST /identity/v3/projects => generated 387 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:49:32.957624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9c1715-2923-4f2b-843d-3aae0059ee30 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:32.967738 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9c1715-2923-4f2b-843d-3aae0059ee30 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:32.971351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:33.024440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214977d7-6868-48b7-9a68-44ae16d85428 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:33.025087 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214977d7-6868-48b7-9a68-44ae16d85428 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.025355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214977d7-6868-48b7-9a68-44ae16d85428 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.025635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-214977d7-6868-48b7-9a68-44ae16d85428 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.026325 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214977d7-6868-48b7-9a68-44ae16d85428 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:33.028279 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-214977d7-6868-48b7-9a68-44ae16d85428 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:33.111262 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1424/2843] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:49:32 2022] POST /identity/v3/users => generated 294 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:49:33.115867 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757c1d51-c4a9-40ec-93bf-7d9a90d6940c admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:33.126482 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-757c1d51-c4a9-40ec-93bf-7d9a90d6940c admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:33.131738 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3075d1b9-29f3-4551-8f01-cce110d06d8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:33.132402 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3075d1b9-29f3-4551-8f01-cce110d06d8e admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.132689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3075d1b9-29f3-4551-8f01-cce110d06d8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.133000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3075d1b9-29f3-4551-8f01-cce110d06d8e admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.133734 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3075d1b9-29f3-4551-8f01-cce110d06d8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:33.135349 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3075d1b9-29f3-4551-8f01-cce110d06d8e admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:33.146404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1420/2844] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:49:33.150654 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214977d7-6868-48b7-9a68-44ae16d85428 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:33.161476 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-214977d7-6868-48b7-9a68-44ae16d85428 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:33.166092 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07952d5-658a-40b2-9986-b88a157bbf93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:33.166742 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07952d5-658a-40b2-9986-b88a157bbf93 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.167027 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07952d5-658a-40b2-9986-b88a157bbf93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.167309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07952d5-658a-40b2-9986-b88a157bbf93 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.168037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07952d5-658a-40b2-9986-b88a157bbf93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:33.169670 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07952d5-658a-40b2-9986-b88a157bbf93 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:33.180202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1425/2845] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jun 10 13:49:33.184286 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3075d1b9-29f3-4551-8f01-cce110d06d8e admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:33.194221 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3075d1b9-29f3-4551-8f01-cce110d06d8e admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:33.199104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8fc427-4696-456f-b02e-1fff9e3a40fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:33.200618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8fc427-4696-456f-b02e-1fff9e3a40fc admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.200618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8fc427-4696-456f-b02e-1fff9e3a40fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.200618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8fc427-4696-456f-b02e-1fff9e3a40fc admin admin] PATH_INFO: `/v3/projects/a1cbdfe6a03047d3a7719ae454b01788/users/e23c321277a84b1daec10f5cd1a7f27e/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.214980 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8fc427-4696-456f-b02e-1fff9e3a40fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1cbdfe6a03047d3a7719ae454b01788, user_id=e23c321277a84b1daec10f5cd1a7f27e, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:33.217668 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8fc427-4696-456f-b02e-1fff9e3a40fc admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:33.305925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1421/2846] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:49:33 2022] PUT /identity/v3/projects/a1cbdfe6a03047d3a7719ae454b01788/users/e23c321277a84b1daec10f5cd1a7f27e/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:33.316118 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a083af-33dd-4fcd-9c2b-d5e2bb943583 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.316413 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a083af-33dd-4fcd-9c2b-d5e2bb943583 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.316699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a083af-33dd-4fcd-9c2b-d5e2bb943583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.343925 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-79a083af-33dd-4fcd-9c2b-d5e2bb943583 None None] MFA Rules not processed for user `e23c321277a84b1daec10f5cd1a7f27e`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:33.345165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-79a083af-33dd-4fcd-9c2b-d5e2bb943583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:33.407830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1426/2847] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:33 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:33.412756 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8fc427-4696-456f-b02e-1fff9e3a40fc admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:33.473119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8fc427-4696-456f-b02e-1fff9e3a40fc admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:33.477717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa673b-e021-48be-ba8e-e16edd830692 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:33.478360 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa673b-e021-48be-ba8e-e16edd830692 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.478643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa673b-e021-48be-ba8e-e16edd830692 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.478919 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76aa673b-e021-48be-ba8e-e16edd830692 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.479601 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa673b-e021-48be-ba8e-e16edd830692 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:33.481536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76aa673b-e021-48be-ba8e-e16edd830692 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:33.563504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1422/2848] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:49:33 2022] POST /identity/v3/projects => generated 387 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:49:33.567475 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a083af-33dd-4fcd-9c2b-d5e2bb943583 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:33.578497 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a083af-33dd-4fcd-9c2b-d5e2bb943583 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:33.634144 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a5070f-f831-4dec-a670-5837c64d2d67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:33.634833 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a5070f-f831-4dec-a670-5837c64d2d67 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.635116 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a5070f-f831-4dec-a670-5837c64d2d67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.635395 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a5070f-f831-4dec-a670-5837c64d2d67 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.636078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a5070f-f831-4dec-a670-5837c64d2d67 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:33.638023 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a5070f-f831-4dec-a670-5837c64d2d67 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:33.696830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1427/2849] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:49:33 2022] POST /identity/v3/users => generated 294 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:49:33.700639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa673b-e021-48be-ba8e-e16edd830692 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:33.711329 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76aa673b-e021-48be-ba8e-e16edd830692 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:33.715994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b8a1ac-2ecd-4cd7-870d-b61d094fc1f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:33.716641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b8a1ac-2ecd-4cd7-870d-b61d094fc1f3 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.716954 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b8a1ac-2ecd-4cd7-870d-b61d094fc1f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.717234 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b8a1ac-2ecd-4cd7-870d-b61d094fc1f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.717958 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b8a1ac-2ecd-4cd7-870d-b61d094fc1f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:33.719615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b8a1ac-2ecd-4cd7-870d-b61d094fc1f3 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:33.720835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:33.730639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1423/2850] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:49:33.740068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a5070f-f831-4dec-a670-5837c64d2d67 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:33.746367 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:33.750098 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a5070f-f831-4dec-a670-5837c64d2d67 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:33.755425 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937c2240-5392-4d62-9c50-b83656c65446 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:33.756062 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937c2240-5392-4d62-9c50-b83656c65446 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.756346 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937c2240-5392-4d62-9c50-b83656c65446 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.756625 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937c2240-5392-4d62-9c50-b83656c65446 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.757371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937c2240-5392-4d62-9c50-b83656c65446 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:33.758962 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937c2240-5392-4d62-9c50-b83656c65446 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:33.768648 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1428/2851] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:33 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:49:33.772486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b8a1ac-2ecd-4cd7-870d-b61d094fc1f3 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:33.782331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b8a1ac-2ecd-4cd7-870d-b61d094fc1f3 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:33.787572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983e8758-a3bc-444f-98e5-f85dcecdb3d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:33.788432 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983e8758-a3bc-444f-98e5-f85dcecdb3d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.788926 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983e8758-a3bc-444f-98e5-f85dcecdb3d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.789062 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983e8758-a3bc-444f-98e5-f85dcecdb3d4 admin admin] PATH_INFO: `/v3/projects/819613c52a144e6aa09adbe7b83a5dbb/users/2ca2ea76982449e198963f69d6cf3332/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.803582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983e8758-a3bc-444f-98e5-f85dcecdb3d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=819613c52a144e6aa09adbe7b83a5dbb, user_id=2ca2ea76982449e198963f69d6cf3332, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:33.806245 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983e8758-a3bc-444f-98e5-f85dcecdb3d4 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:33.890635 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1424/2852] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:49:33 2022] PUT /identity/v3/projects/819613c52a144e6aa09adbe7b83a5dbb/users/2ca2ea76982449e198963f69d6cf3332/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:33.933946 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282a182e-650b-4190-b593-fbba5aa4217b None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:33.934261 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282a182e-650b-4190-b593-fbba5aa4217b None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:33.934550 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282a182e-650b-4190-b593-fbba5aa4217b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:33.960976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-282a182e-650b-4190-b593-fbba5aa4217b None None] MFA Rules not processed for user `2ca2ea76982449e198963f69d6cf3332`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:33.962199 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-282a182e-650b-4190-b593-fbba5aa4217b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:34.026271 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1429/2853] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:33 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:34.030628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983e8758-a3bc-444f-98e5-f85dcecdb3d4 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:34.089979 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983e8758-a3bc-444f-98e5-f85dcecdb3d4 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:34.094545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c70ed1-f3e1-4ddb-b077-171d17d461e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:34.095190 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c70ed1-f3e1-4ddb-b077-171d17d461e3 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:34.095473 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c70ed1-f3e1-4ddb-b077-171d17d461e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:34.095748 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c70ed1-f3e1-4ddb-b077-171d17d461e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:34.096440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c70ed1-f3e1-4ddb-b077-171d17d461e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:34.098904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3c70ed1-f3e1-4ddb-b077-171d17d461e3 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:34.188397 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1425/2854] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:49:34 2022] POST /identity/v3/projects => generated 385 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:49:34.192471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282a182e-650b-4190-b593-fbba5aa4217b None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:34.203460 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282a182e-650b-4190-b593-fbba5aa4217b None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:34.258932 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87548150-2340-4461-9335-a5cf88cefb79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:34.259619 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87548150-2340-4461-9335-a5cf88cefb79 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:34.259906 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87548150-2340-4461-9335-a5cf88cefb79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:34.260190 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87548150-2340-4461-9335-a5cf88cefb79 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:34.260919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87548150-2340-4461-9335-a5cf88cefb79 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:34.262883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87548150-2340-4461-9335-a5cf88cefb79 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:34.346177 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1430/2855] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:49:34 2022] POST /identity/v3/users => generated 299 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:49:34.350854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c70ed1-f3e1-4ddb-b077-171d17d461e3 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:34.361028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c70ed1-f3e1-4ddb-b077-171d17d461e3 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:34.365882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ff962c-b48d-46e1-a490-5270654e4377 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:34.366536 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ff962c-b48d-46e1-a490-5270654e4377 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:34.366817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ff962c-b48d-46e1-a490-5270654e4377 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:34.367095 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ff962c-b48d-46e1-a490-5270654e4377 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:34.367820 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ff962c-b48d-46e1-a490-5270654e4377 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:34.369462 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3ff962c-b48d-46e1-a490-5270654e4377 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:34.379607 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1426/2856] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:34 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:49:34.383484 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87548150-2340-4461-9335-a5cf88cefb79 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:34.393414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87548150-2340-4461-9335-a5cf88cefb79 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:34.398488 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31030a0-21a7-4c25-b3d3-a13c260445e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:34.399354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31030a0-21a7-4c25-b3d3-a13c260445e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:34.399637 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31030a0-21a7-4c25-b3d3-a13c260445e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:34.399919 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31030a0-21a7-4c25-b3d3-a13c260445e9 admin admin] PATH_INFO: `/v3/projects/b8d7d74a9f1c49c794fe6bc2b34fe9b9/users/632a33b47b5c4e1d8d93e3c524935df2/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:34.414945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31030a0-21a7-4c25-b3d3-a13c260445e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8d7d74a9f1c49c794fe6bc2b34fe9b9, user_id=632a33b47b5c4e1d8d93e3c524935df2, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:34.417655 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31030a0-21a7-4c25-b3d3-a13c260445e9 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:34.509077 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1431/2857] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:49:34 2022] PUT /identity/v3/projects/b8d7d74a9f1c49c794fe6bc2b34fe9b9/users/632a33b47b5c4e1d8d93e3c524935df2/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:34.513213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ff962c-b48d-46e1-a490-5270654e4377 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:34.527235 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3ff962c-b48d-46e1-a490-5270654e4377 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:34.583405 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6355c01-c534-4d2f-b20c-b7b9390358a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:34.584057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6355c01-c534-4d2f-b20c-b7b9390358a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:34.584455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6355c01-c534-4d2f-b20c-b7b9390358a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:34.584736 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6355c01-c534-4d2f-b20c-b7b9390358a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:34.585518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6355c01-c534-4d2f-b20c-b7b9390358a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:34.587129 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6355c01-c534-4d2f-b20c-b7b9390358a2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:34.596632 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1427/2858] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:34 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:49:34.600882 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31030a0-21a7-4c25-b3d3-a13c260445e9 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:34.611158 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31030a0-21a7-4c25-b3d3-a13c260445e9 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:34.616333 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e907dd12-78e1-48df-97bc-bcb6949d5ef2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:34.617240 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e907dd12-78e1-48df-97bc-bcb6949d5ef2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:34.617528 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e907dd12-78e1-48df-97bc-bcb6949d5ef2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:34.617805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e907dd12-78e1-48df-97bc-bcb6949d5ef2 admin admin] PATH_INFO: `/v3/projects/b8d7d74a9f1c49c794fe6bc2b34fe9b9/users/632a33b47b5c4e1d8d93e3c524935df2/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:34.619749 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e907dd12-78e1-48df-97bc-bcb6949d5ef2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8d7d74a9f1c49c794fe6bc2b34fe9b9, user_id=632a33b47b5c4e1d8d93e3c524935df2, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:34.622427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e907dd12-78e1-48df-97bc-bcb6949d5ef2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:34.633135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1432/2859] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:49:34 2022] PUT /identity/v3/projects/b8d7d74a9f1c49c794fe6bc2b34fe9b9/users/632a33b47b5c4e1d8d93e3c524935df2/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:34.644011 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065d658a-324b-48b4-89b8-6d51cae4c5a9 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:34.644308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065d658a-324b-48b4-89b8-6d51cae4c5a9 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:34.644598 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065d658a-324b-48b4-89b8-6d51cae4c5a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:34.670378 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-065d658a-324b-48b4-89b8-6d51cae4c5a9 None None] MFA Rules not processed for user `632a33b47b5c4e1d8d93e3c524935df2`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:34.671578 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-065d658a-324b-48b4-89b8-6d51cae4c5a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:34.700251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:34.737239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1428/2860] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:34 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:34.748529 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e907dd12-78e1-48df-97bc-bcb6949d5ef2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:34.750273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-e907dd12-78e1-48df-97bc-bcb6949d5ef2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:34.789974 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:34.816525 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e907dd12-78e1-48df-97bc-bcb6949d5ef2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:34.820688 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8049b414-a133-4034-9d23-8106249776cd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:34.821363 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8049b414-a133-4034-9d23-8106249776cd service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:34.821731 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8049b414-a133-4034-9d23-8106249776cd service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:34.822020 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8049b414-a133-4034-9d23-8106249776cd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:34.831176 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8049b414-a133-4034-9d23-8106249776cd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:34.832643 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8049b414-a133-4034-9d23-8106249776cd service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:34.842412 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1433/2861] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:49:34 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:49:35.895006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065d658a-324b-48b4-89b8-6d51cae4c5a9 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:35.896323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-065d658a-324b-48b4-89b8-6d51cae4c5a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:35.907940 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065d658a-324b-48b4-89b8-6d51cae4c5a9 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:35.911565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191129df-290d-4862-8c52-6a4af624b1c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:35.912195 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191129df-290d-4862-8c52-6a4af624b1c3 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:35.912476 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191129df-290d-4862-8c52-6a4af624b1c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:35.912751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191129df-290d-4862-8c52-6a4af624b1c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:35.967589 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191129df-290d-4862-8c52-6a4af624b1c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:35.969049 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191129df-290d-4862-8c52-6a4af624b1c3 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:35.977713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1429/2862] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:49:35 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:49:40.730660 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8049b414-a133-4034-9d23-8106249776cd service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:40.739298 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8049b414-a133-4034-9d23-8106249776cd service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:40.742945 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba8ee2-2cfa-4b41-8837-9145c175173a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:40.743581 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba8ee2-2cfa-4b41-8837-9145c175173a service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:40.743976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba8ee2-2cfa-4b41-8837-9145c175173a service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:40.744140 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaba8ee2-2cfa-4b41-8837-9145c175173a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:40.801128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaba8ee2-2cfa-4b41-8837-9145c175173a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:40.802578 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaba8ee2-2cfa-4b41-8837-9145c175173a service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:40.811375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1434/2863] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:49:40 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:49:43.450617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191129df-290d-4862-8c52-6a4af624b1c3 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:43.452304 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-191129df-290d-4862-8c52-6a4af624b1c3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:43.530831 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191129df-290d-4862-8c52-6a4af624b1c3 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:43.536643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b177c94-e18b-4c65-aa3d-153e8e82372e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:43.537307 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b177c94-e18b-4c65-aa3d-153e8e82372e service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:43.548606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b177c94-e18b-4c65-aa3d-153e8e82372e service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:43.548606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b177c94-e18b-4c65-aa3d-153e8e82372e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:43.548606 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9b177c94-e18b-4c65-aa3d-153e8e82372e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:43.550955 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b177c94-e18b-4c65-aa3d-153e8e82372e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:43.552514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b177c94-e18b-4c65-aa3d-153e8e82372e service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:43.562159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1430/2864] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:49:43 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:49:44.526405 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba8ee2-2cfa-4b41-8837-9145c175173a service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:44.587151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaba8ee2-2cfa-4b41-8837-9145c175173a service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:44.591711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:44.592564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:44.592875 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:44.593151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] PATH_INFO: `/v3/users/e23c321277a84b1daec10f5cd1a7f27e` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:44.594318 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e23c321277a84b1daec10f5cd1a7f27e)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:44.596309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:44.815873 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e23c321277a84b1daec10f5cd1a7f27e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:44.816533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e23c321277a84b1daec10f5cd1a7f27e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:44.817174 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e23c321277a84b1daec10f5cd1a7f27e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:44.824057 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e23c321277a84b1daec10f5cd1a7f27e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:44.882520 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e23c321277a84b1daec10f5cd1a7f27e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:44.887673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:44.900156 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1435/2865] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:49:44 2022] DELETE /identity/v3/users/e23c321277a84b1daec10f5cd1a7f27e => generated 0 bytes in 374 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:44.904654 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53069bc6-9bdc-4ff7-aaf6-43b5e091bec8 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:44.905148 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53069bc6-9bdc-4ff7-aaf6-43b5e091bec8 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:44.905251 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53069bc6-9bdc-4ff7-aaf6-43b5e091bec8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:44.926466 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-53069bc6-9bdc-4ff7-aaf6-43b5e091bec8 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:44.927671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-53069bc6-9bdc-4ff7-aaf6-43b5e091bec8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:44.969491 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=100 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:44.986070 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1431/2866] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:44.996233 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:44.997635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:45.053519 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86dbb56c-8d03-4706-b358-727aab20dac9 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:45.057186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795961fe-cbb5-4725-8336-2b32030036ae service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:45.057828 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795961fe-cbb5-4725-8336-2b32030036ae service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:45.058114 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795961fe-cbb5-4725-8336-2b32030036ae service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:45.058396 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-795961fe-cbb5-4725-8336-2b32030036ae service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:45.068160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795961fe-cbb5-4725-8336-2b32030036ae service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:45.069632 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-795961fe-cbb5-4725-8336-2b32030036ae service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:45.079302 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1436/2867] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:49:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:49:45.503136 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53069bc6-9bdc-4ff7-aaf6-43b5e091bec8 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:45.505318 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-53069bc6-9bdc-4ff7-aaf6-43b5e091bec8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:45.517453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53069bc6-9bdc-4ff7-aaf6-43b5e091bec8 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:45.522711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:45.523570 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:45.523851 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:45.524132 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] PATH_INFO: `/v3/projects/a1cbdfe6a03047d3a7719ae454b01788` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:45.525346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1cbdfe6a03047d3a7719ae454b01788)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:45.527254 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:45.636292 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1cbdfe6a03047d3a7719ae454b01788', 'request_id': 'req-117a5e18-8d01-44fa-995b-2ef335390caf'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:45.642200 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1cbdfe6a03047d3a7719ae454b01788', 'request_id': 'req-117a5e18-8d01-44fa-995b-2ef335390caf'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:45.648120 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1cbdfe6a03047d3a7719ae454b01788', 'request_id': 'req-117a5e18-8d01-44fa-995b-2ef335390caf'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:45.715636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] The token cache is being invalidate because project a1cbdfe6a03047d3a7719ae454b01788 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:49:45.715906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:45.716520 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:45.718061 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1432/2868] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:49:45 2022] DELETE /identity/v3/projects/a1cbdfe6a03047d3a7719ae454b01788 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:45.722324 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795961fe-cbb5-4725-8336-2b32030036ae service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:45.732597 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-795961fe-cbb5-4725-8336-2b32030036ae service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:45.790506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:45.791762 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:45.792051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:45.792334 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] PATH_INFO: `/v3/users/2ca2ea76982449e198963f69d6cf3332` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:45.793565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ca2ea76982449e198963f69d6cf3332)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:45.795533 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:46.033630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ca2ea76982449e198963f69d6cf3332'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.034300 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ca2ea76982449e198963f69d6cf3332'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.034917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ca2ea76982449e198963f69d6cf3332'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.044224 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ca2ea76982449e198963f69d6cf3332'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.119732 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ca2ea76982449e198963f69d6cf3332'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.137706 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1437/2869] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:49:45 2022] DELETE /identity/v3/users/2ca2ea76982449e198963f69d6cf3332 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:46.542042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:46.543488 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:46.556411 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-117a5e18-8d01-44fa-995b-2ef335390caf admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:46.611816 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:46.612686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:46.612993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:46.613268 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] PATH_INFO: `/v3/projects/819613c52a144e6aa09adbe7b83a5dbb` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:46.614440 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] RBAC: Authorizing `identity:delete_project(project_id=819613c52a144e6aa09adbe7b83a5dbb)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:46.616330 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:46.730588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '819613c52a144e6aa09adbe7b83a5dbb', 'request_id': 'req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.736645 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '819613c52a144e6aa09adbe7b83a5dbb', 'request_id': 'req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.742308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '819613c52a144e6aa09adbe7b83a5dbb', 'request_id': 'req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.804545 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:46.808367 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] The token cache is being invalidate because project 819613c52a144e6aa09adbe7b83a5dbb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:49:46.808600 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.809248 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:46.810832 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1433/2870] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:49:46 2022] DELETE /identity/v3/projects/819613c52a144e6aa09adbe7b83a5dbb => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:46.814897 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:46.816287 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:46.829844 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861b385c-7f3e-4db5-b57e-a072ac6b522c admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:46.834991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:46.885165 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:46.886029 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:46.886310 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:46.886585 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] PATH_INFO: `/v3/users/632a33b47b5c4e1d8d93e3c524935df2` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:46.887747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=632a33b47b5c4e1d8d93e3c524935df2)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:46.889779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:47.110772 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '632a33b47b5c4e1d8d93e3c524935df2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.111341 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '632a33b47b5c4e1d8d93e3c524935df2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.111942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '632a33b47b5c4e1d8d93e3c524935df2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.118927 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '632a33b47b5c4e1d8d93e3c524935df2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.205355 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-d88e2e51-0ab1-48c8-9d6f-fcd001efb9c9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '632a33b47b5c4e1d8d93e3c524935df2'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.222666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1438/2871] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:49:46 2022] DELETE /identity/v3/users/632a33b47b5c4e1d8d93e3c524935df2 => generated 0 bytes in 408 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:49:47.634824 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:47.636523 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:47.650698 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a8e0a9-fc88-4dc5-8055-5c3eb8ab492f admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:47.706625 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:47.707484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:47.707767 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:47.708047 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] PATH_INFO: `/v3/projects/b8d7d74a9f1c49c794fe6bc2b34fe9b9` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:47.709258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8d7d74a9f1c49c794fe6bc2b34fe9b9)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:47.711167 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:47.815869 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8d7d74a9f1c49c794fe6bc2b34fe9b9', 'request_id': 'req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.821796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8d7d74a9f1c49c794fe6bc2b34fe9b9', 'request_id': 'req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.827557 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8d7d74a9f1c49c794fe6bc2b34fe9b9', 'request_id': 'req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.901240 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] The token cache is being invalidate because project b8d7d74a9f1c49c794fe6bc2b34fe9b9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:49:47.901467 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.902082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-9b847b9a-b9af-4bee-857f-b1a74f4a7dc8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:49:47.903618 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1434/2872] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:49:47 2022] DELETE /identity/v3/projects/b8d7d74a9f1c49c794fe6bc2b34fe9b9 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:47.909509 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e7969-55d5-4ead-af68-0a9da4a0c928 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:47.909999 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e7969-55d5-4ead-af68-0a9da4a0c928 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:47.910086 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae8e7969-55d5-4ead-af68-0a9da4a0c928 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:47.927763 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ae8e7969-55d5-4ead-af68-0a9da4a0c928 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:47.929011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ae8e7969-55d5-4ead-af68-0a9da4a0c928 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:47.986453 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1439/2873] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:47.994699 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d301d2b-41a7-49cc-9f31-3e7c7f587489 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:47.994992 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d301d2b-41a7-49cc-9f31-3e7c7f587489 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:47.995287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d301d2b-41a7-49cc-9f31-3e7c7f587489 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.013866 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-8d301d2b-41a7-49cc-9f31-3e7c7f587489 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:48.015058 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8d301d2b-41a7-49cc-9f31-3e7c7f587489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:48.023293 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1435/2874] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:48.027206 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e7969-55d5-4ead-af68-0a9da4a0c928 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:48.037295 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae8e7969-55d5-4ead-af68-0a9da4a0c928 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:48.041799 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bea923b-e6bf-4de8-8045-9e0609f3ed93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:48.042440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bea923b-e6bf-4de8-8045-9e0609f3ed93 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.042718 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bea923b-e6bf-4de8-8045-9e0609f3ed93 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.042990 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bea923b-e6bf-4de8-8045-9e0609f3ed93 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.043700 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bea923b-e6bf-4de8-8045-9e0609f3ed93 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:49:48.044025 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bea923b-e6bf-4de8-8045-9e0609f3ed93 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:48.045631 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bea923b-e6bf-4de8-8045-9e0609f3ed93 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:48.057169 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1440/2875] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:49:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:49:48.061243 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d301d2b-41a7-49cc-9f31-3e7c7f587489 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:48.070078 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d301d2b-41a7-49cc-9f31-3e7c7f587489 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:48.074573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f674f7a2-bebb-4fe6-861b-eb34bcad9b45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:48.075207 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f674f7a2-bebb-4fe6-861b-eb34bcad9b45 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.075489 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f674f7a2-bebb-4fe6-861b-eb34bcad9b45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.075767 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f674f7a2-bebb-4fe6-861b-eb34bcad9b45 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.076714 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f674f7a2-bebb-4fe6-861b-eb34bcad9b45 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:48.078621 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f674f7a2-bebb-4fe6-861b-eb34bcad9b45 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:48.196030 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1436/2876] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:49:48 2022] POST /identity/v3/projects => generated 385 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:49:48.200142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bea923b-e6bf-4de8-8045-9e0609f3ed93 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:48.211427 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bea923b-e6bf-4de8-8045-9e0609f3ed93 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:48.222829 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:48.267311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18152d-8713-475f-a15f-2dddaf9d921d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:48.267979 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18152d-8713-475f-a15f-2dddaf9d921d admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.268270 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18152d-8713-475f-a15f-2dddaf9d921d admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.268553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b18152d-8713-475f-a15f-2dddaf9d921d admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.269273 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b18152d-8713-475f-a15f-2dddaf9d921d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:48.271863 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b18152d-8713-475f-a15f-2dddaf9d921d admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:48.359890 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1441/2877] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:49:48 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:49:48.364184 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f674f7a2-bebb-4fe6-861b-eb34bcad9b45 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:48.374504 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f674f7a2-bebb-4fe6-861b-eb34bcad9b45 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:48.379704 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1b4d56-ad89-4bdb-ac8b-05f63d6a70dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:48.380346 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1b4d56-ad89-4bdb-ac8b-05f63d6a70dd admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.380626 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1b4d56-ad89-4bdb-ac8b-05f63d6a70dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.380936 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1b4d56-ad89-4bdb-ac8b-05f63d6a70dd admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.381737 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1b4d56-ad89-4bdb-ac8b-05f63d6a70dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:48.383305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1b4d56-ad89-4bdb-ac8b-05f63d6a70dd admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:48.393494 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1437/2878] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:49:48.397818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18152d-8713-475f-a15f-2dddaf9d921d admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:48.407769 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b18152d-8713-475f-a15f-2dddaf9d921d admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:48.412373 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fbf9bd-ada6-4796-95d1-c322969dd2c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:48.413088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fbf9bd-ada6-4796-95d1-c322969dd2c5 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.413378 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fbf9bd-ada6-4796-95d1-c322969dd2c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.413658 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fbf9bd-ada6-4796-95d1-c322969dd2c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.414381 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fbf9bd-ada6-4796-95d1-c322969dd2c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:48.415968 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fbf9bd-ada6-4796-95d1-c322969dd2c5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:48.425712 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1442/2879] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:48 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:49:48.429586 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1b4d56-ad89-4bdb-ac8b-05f63d6a70dd admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:48.439287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1b4d56-ad89-4bdb-ac8b-05f63d6a70dd admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:48.445309 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801948b7-58a1-4fb3-b2ec-629929394bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:48.446717 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801948b7-58a1-4fb3-b2ec-629929394bfb admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.447157 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801948b7-58a1-4fb3-b2ec-629929394bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.447636 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-801948b7-58a1-4fb3-b2ec-629929394bfb admin admin] PATH_INFO: `/v3/projects/7f7b453571c14951bee74adae4259f1a/users/ee32d0817496444c8b7c6dcc63bae246/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.465916 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801948b7-58a1-4fb3-b2ec-629929394bfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f7b453571c14951bee74adae4259f1a, user_id=ee32d0817496444c8b7c6dcc63bae246, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:48.468596 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-801948b7-58a1-4fb3-b2ec-629929394bfb admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:48.547801 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1438/2880] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:49:48 2022] PUT /identity/v3/projects/7f7b453571c14951bee74adae4259f1a/users/ee32d0817496444c8b7c6dcc63bae246/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:48.559296 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07dc71-72e9-41af-a83c-1dd32192d680 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.559590 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07dc71-72e9-41af-a83c-1dd32192d680 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.559902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea07dc71-72e9-41af-a83c-1dd32192d680 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.586251 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ea07dc71-72e9-41af-a83c-1dd32192d680 None None] MFA Rules not processed for user `ee32d0817496444c8b7c6dcc63bae246`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:48.587449 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ea07dc71-72e9-41af-a83c-1dd32192d680 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:48.649351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1443/2881] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:48 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:48.653713 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801948b7-58a1-4fb3-b2ec-629929394bfb admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:48.682874 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:48.713434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-801948b7-58a1-4fb3-b2ec-629929394bfb admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:48.718486 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2620d10-b5fe-4866-8a4c-bd307eadfe38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:48.720313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2620d10-b5fe-4866-8a4c-bd307eadfe38 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.720313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2620d10-b5fe-4866-8a4c-bd307eadfe38 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.720313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2620d10-b5fe-4866-8a4c-bd307eadfe38 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.720313 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2620d10-b5fe-4866-8a4c-bd307eadfe38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:48.722226 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2620d10-b5fe-4866-8a4c-bd307eadfe38 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:48.806422 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1439/2882] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:49:48 2022] POST /identity/v3/projects => generated 385 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:49:48.810630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea07dc71-72e9-41af-a83c-1dd32192d680 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:48.821231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea07dc71-72e9-41af-a83c-1dd32192d680 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:48.879691 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa59cdf-a441-46f8-abba-b7d8bdf3ef76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:48.880356 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa59cdf-a441-46f8-abba-b7d8bdf3ef76 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.880632 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa59cdf-a441-46f8-abba-b7d8bdf3ef76 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.881031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa59cdf-a441-46f8-abba-b7d8bdf3ef76 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:48.881721 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa59cdf-a441-46f8-abba-b7d8bdf3ef76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:48.883638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa59cdf-a441-46f8-abba-b7d8bdf3ef76 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:48.980218 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1444/2883] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:49:48 2022] POST /identity/v3/users => generated 299 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:49:48.983948 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2620d10-b5fe-4866-8a4c-bd307eadfe38 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:48.994224 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2620d10-b5fe-4866-8a4c-bd307eadfe38 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:48.998754 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ba020-917a-4831-a60f-716f833f7375 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:48.999400 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ba020-917a-4831-a60f-716f833f7375 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:48.999684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ba020-917a-4831-a60f-716f833f7375 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:48.999966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6ba020-917a-4831-a60f-716f833f7375 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:49.000689 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ba020-917a-4831-a60f-716f833f7375 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:49.002303 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6ba020-917a-4831-a60f-716f833f7375 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:49.013083 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1440/2884] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:48 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:49:49.017079 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa59cdf-a441-46f8-abba-b7d8bdf3ef76 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:49.027789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa59cdf-a441-46f8-abba-b7d8bdf3ef76 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:49.032380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb1585a-238d-4bff-a375-39c538c1acb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:49.033289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb1585a-238d-4bff-a375-39c538c1acb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:49.033576 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb1585a-238d-4bff-a375-39c538c1acb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:49.033853 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb1585a-238d-4bff-a375-39c538c1acb2 admin admin] PATH_INFO: `/v3/projects/b5f2f879fccf4628993f6024f9ea15f3/users/c4caec900517407b95e22bb2eb09bcb4/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:49.044944 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:49:49.048699 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb1585a-238d-4bff-a375-39c538c1acb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5f2f879fccf4628993f6024f9ea15f3, user_id=c4caec900517407b95e22bb2eb09bcb4, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:49.051376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb1585a-238d-4bff-a375-39c538c1acb2 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:49.165789 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1445/2885] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:49:49 2022] PUT /identity/v3/projects/b5f2f879fccf4628993f6024f9ea15f3/users/c4caec900517407b95e22bb2eb09bcb4/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:49.169798 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ba020-917a-4831-a60f-716f833f7375 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:49.179675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6ba020-917a-4831-a60f-716f833f7375 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:49.234710 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d972ea-c07c-4265-943e-c5d99fe9c092 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:49.235308 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d972ea-c07c-4265-943e-c5d99fe9c092 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:49.235594 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d972ea-c07c-4265-943e-c5d99fe9c092 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:49.235875 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d972ea-c07c-4265-943e-c5d99fe9c092 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:49.236597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d972ea-c07c-4265-943e-c5d99fe9c092 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:49.238221 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d972ea-c07c-4265-943e-c5d99fe9c092 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:49.248325 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1441/2886] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:49:49 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:49:49.252547 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb1585a-238d-4bff-a375-39c538c1acb2 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:49.263601 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb1585a-238d-4bff-a375-39c538c1acb2 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:49.268245 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2521ee96-8e22-4a8e-9693-b08fc5b3a69c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:49.269151 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2521ee96-8e22-4a8e-9693-b08fc5b3a69c admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:49.269438 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2521ee96-8e22-4a8e-9693-b08fc5b3a69c admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:49.269758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2521ee96-8e22-4a8e-9693-b08fc5b3a69c admin admin] PATH_INFO: `/v3/projects/b5f2f879fccf4628993f6024f9ea15f3/users/c4caec900517407b95e22bb2eb09bcb4/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:49.271708 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2521ee96-8e22-4a8e-9693-b08fc5b3a69c admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5f2f879fccf4628993f6024f9ea15f3, user_id=c4caec900517407b95e22bb2eb09bcb4, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:49.274404 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2521ee96-8e22-4a8e-9693-b08fc5b3a69c admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:49.284558 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1446/2887] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:49:49 2022] PUT /identity/v3/projects/b5f2f879fccf4628993f6024f9ea15f3/users/c4caec900517407b95e22bb2eb09bcb4/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:49:49.295595 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad225d9-b585-4c51-b459-37b19af42669 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:49.295961 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad225d9-b585-4c51-b459-37b19af42669 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:49.296331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ad225d9-b585-4c51-b459-37b19af42669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:49.323430 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-2ad225d9-b585-4c51-b459-37b19af42669 None None] MFA Rules not processed for user `c4caec900517407b95e22bb2eb09bcb4`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:49:49.324641 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2ad225d9-b585-4c51-b459-37b19af42669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:49.392000 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1442/2888] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:49:49 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:49:49.402782 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2521ee96-8e22-4a8e-9693-b08fc5b3a69c admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:49.404468 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-2521ee96-8e22-4a8e-9693-b08fc5b3a69c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:49.460552 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2521ee96-8e22-4a8e-9693-b08fc5b3a69c admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:49.464177 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c046c78-d9f1-4830-b665-05497dee7e4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:49.464842 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c046c78-d9f1-4830-b665-05497dee7e4a service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:49.465133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c046c78-d9f1-4830-b665-05497dee7e4a service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:49.465414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c046c78-d9f1-4830-b665-05497dee7e4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:49.526673 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c046c78-d9f1-4830-b665-05497dee7e4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:49.528051 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c046c78-d9f1-4830-b665-05497dee7e4a service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:49.536758 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1447/2889] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:49:49 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:49:54.447086 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad225d9-b585-4c51-b459-37b19af42669 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:49:54.448460 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-2ad225d9-b585-4c51-b459-37b19af42669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:49:54.460883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ad225d9-b585-4c51-b459-37b19af42669 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:49:54.464353 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf67732a-fd72-4bc7-8ee7-5205f4a2114c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:49:54.465077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67732a-fd72-4bc7-8ee7-5205f4a2114c service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:49:54.465363 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67732a-fd72-4bc7-8ee7-5205f4a2114c service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:49:54.465639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf67732a-fd72-4bc7-8ee7-5205f4a2114c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:49:54.474858 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf67732a-fd72-4bc7-8ee7-5205f4a2114c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:49:54.476276 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf67732a-fd72-4bc7-8ee7-5205f4a2114c service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:49:54.485445 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1443/2890] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:49:54 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:49:59.532934 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=57,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:00.484292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c046c78-d9f1-4830-b665-05497dee7e4a service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:00.551885 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c046c78-d9f1-4830-b665-05497dee7e4a service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:00.558130 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:00.559390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:00.559390 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:00.559586 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] PATH_INFO: `/v3/users/ee32d0817496444c8b7c6dcc63bae246` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:00.560747 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee32d0817496444c8b7c6dcc63bae246)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:00.562717 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:00.733904 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee32d0817496444c8b7c6dcc63bae246'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:00.734482 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee32d0817496444c8b7c6dcc63bae246'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:00.735101 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee32d0817496444c8b7c6dcc63bae246'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:00.742477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee32d0817496444c8b7c6dcc63bae246'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:00.825971 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee32d0817496444c8b7c6dcc63bae246'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:00.844598 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1448/2891] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:50:00 2022] DELETE /identity/v3/users/ee32d0817496444c8b7c6dcc63bae246 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:00.849203 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c494379-ec46-4e45-940c-402ca5c07d29 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:00.849685 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c494379-ec46-4e45-940c-402ca5c07d29 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:00.849785 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c494379-ec46-4e45-940c-402ca5c07d29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:00.866804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-6c494379-ec46-4e45-940c-402ca5c07d29 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:00.868006 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6c494379-ec46-4e45-940c-402ca5c07d29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:00.873442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:00.944305 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1444/2892] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:00.954064 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:00.955448 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:01.012088 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a30f97f-0547-469c-b2ba-201021ff72ba admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:01.015629 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c352688-7770-42f9-988b-a9923abeab72 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:01.016267 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c352688-7770-42f9-988b-a9923abeab72 service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:01.016553 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c352688-7770-42f9-988b-a9923abeab72 service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:01.016862 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c352688-7770-42f9-988b-a9923abeab72 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:01.026292 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c352688-7770-42f9-988b-a9923abeab72 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:01.027720 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c352688-7770-42f9-988b-a9923abeab72 service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:01.037119 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1449/2893] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:50:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:50:01.427469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c494379-ec46-4e45-940c-402ca5c07d29 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:01.428942 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-6c494379-ec46-4e45-940c-402ca5c07d29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:01.441174 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c494379-ec46-4e45-940c-402ca5c07d29 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:01.446442 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:01.447302 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:01.447582 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:01.447860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] PATH_INFO: `/v3/projects/7f7b453571c14951bee74adae4259f1a` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:01.449108 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f7b453571c14951bee74adae4259f1a)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:01.451272 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:01.552671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f7b453571c14951bee74adae4259f1a', 'request_id': 'req-8cfe6d62-4181-447d-aa60-efa0c7b61c24'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:01.559229 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f7b453571c14951bee74adae4259f1a', 'request_id': 'req-8cfe6d62-4181-447d-aa60-efa0c7b61c24'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:01.564966 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f7b453571c14951bee74adae4259f1a', 'request_id': 'req-8cfe6d62-4181-447d-aa60-efa0c7b61c24'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:01.719317 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] The token cache is being invalidate because project 7f7b453571c14951bee74adae4259f1a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:50:01.719605 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:01.720250 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:01.721804 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1445/2894] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:50:01 2022] DELETE /identity/v3/projects/7f7b453571c14951bee74adae4259f1a => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:50:01.726212 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c352688-7770-42f9-988b-a9923abeab72 service neutron] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:01.736078 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c352688-7770-42f9-988b-a9923abeab72 service neutron] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:01.756159 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:01.794408 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:01.795634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:01.795969 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:01.796252 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] PATH_INFO: `/v3/users/c4caec900517407b95e22bb2eb09bcb4` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:01.797471 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c4caec900517407b95e22bb2eb09bcb4)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:01.799459 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:01.944841 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c4caec900517407b95e22bb2eb09bcb4'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:01.945511 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c4caec900517407b95e22bb2eb09bcb4'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:01.946136 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c4caec900517407b95e22bb2eb09bcb4'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:01.953465 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c4caec900517407b95e22bb2eb09bcb4'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:02.042565 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-477c0516-71a4-4dff-9fcb-124768d4de68 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c4caec900517407b95e22bb2eb09bcb4'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:02.059942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1450/2895] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:50:01 2022] DELETE /identity/v3/users/c4caec900517407b95e22bb2eb09bcb4 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:02.429707 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:02.431112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:02.444998 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cfe6d62-4181-447d-aa60-efa0c7b61c24 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:02.502252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:02.503113 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:02.503404 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:02.503706 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] PATH_INFO: `/v3/projects/b5f2f879fccf4628993f6024f9ea15f3` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:02.505423 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5f2f879fccf4628993f6024f9ea15f3)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:02.507337 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:02.607173 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:02.616339 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5f2f879fccf4628993f6024f9ea15f3', 'request_id': 'req-bcc7b401-d8e9-4777-a96d-50c0f9860550'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:02.622323 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5f2f879fccf4628993f6024f9ea15f3', 'request_id': 'req-bcc7b401-d8e9-4777-a96d-50c0f9860550'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:02.627925 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5f2f879fccf4628993f6024f9ea15f3', 'request_id': 'req-bcc7b401-d8e9-4777-a96d-50c0f9860550'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:02.718684 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] The token cache is being invalidate because project b5f2f879fccf4628993f6024f9ea15f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:50:02.718968 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:02.719574 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-bcc7b401-d8e9-4777-a96d-50c0f9860550 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:02.721177 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1446/2896] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:50:02 2022] DELETE /identity/v3/projects/b5f2f879fccf4628993f6024f9ea15f3 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:02.727095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a85df-a0de-4de0-8fae-3f1157479ae7 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:02.728582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a85df-a0de-4de0-8fae-3f1157479ae7 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:02.728582 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5a85df-a0de-4de0-8fae-3f1157479ae7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:02.745608 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-1d5a85df-a0de-4de0-8fae-3f1157479ae7 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:02.746814 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1d5a85df-a0de-4de0-8fae-3f1157479ae7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:02.809093 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1451/2897] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:02.817434 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deebb26a-ce0f-4571-8c78-d1544d1384ec None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:02.818301 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deebb26a-ce0f-4571-8c78-d1544d1384ec None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:02.818588 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deebb26a-ce0f-4571-8c78-d1544d1384ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:02.836485 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-deebb26a-ce0f-4571-8c78-d1544d1384ec None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:02.837719 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-deebb26a-ce0f-4571-8c78-d1544d1384ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:02.845474 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1447/2898] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:02.849426 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5a85df-a0de-4de0-8fae-3f1157479ae7 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:02.858938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5a85df-a0de-4de0-8fae-3f1157479ae7 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:02.863506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afea31-67cf-422c-ab51-81eadfdadd07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:02.864157 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afea31-67cf-422c-ab51-81eadfdadd07 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:02.865011 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afea31-67cf-422c-ab51-81eadfdadd07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:02.865298 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81afea31-67cf-422c-ab51-81eadfdadd07 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:02.866021 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afea31-67cf-422c-ab51-81eadfdadd07 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:50:02.866332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afea31-67cf-422c-ab51-81eadfdadd07 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:02.867938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81afea31-67cf-422c-ab51-81eadfdadd07 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:02.879440 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1452/2899] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:50:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:50:02.884189 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deebb26a-ce0f-4571-8c78-d1544d1384ec None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:02.893155 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deebb26a-ce0f-4571-8c78-d1544d1384ec None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:02.897696 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7669d159-3b32-4777-a470-caf02501fbcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:02.898331 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7669d159-3b32-4777-a470-caf02501fbcd admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:02.898617 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7669d159-3b32-4777-a470-caf02501fbcd admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:02.898893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7669d159-3b32-4777-a470-caf02501fbcd admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:02.899572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7669d159-3b32-4777-a470-caf02501fbcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:02.901524 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7669d159-3b32-4777-a470-caf02501fbcd admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:03.015296 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1448/2900] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:50:02 2022] POST /identity/v3/projects => generated 373 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:50:03.019043 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afea31-67cf-422c-ab51-81eadfdadd07 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:03.028963 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81afea31-67cf-422c-ab51-81eadfdadd07 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:03.054794 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:03.084664 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6d53de-03d4-4466-aaec-837de790e92a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:03.085354 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6d53de-03d4-4466-aaec-837de790e92a admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:03.085638 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6d53de-03d4-4466-aaec-837de790e92a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:03.085917 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6d53de-03d4-4466-aaec-837de790e92a admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:03.086626 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6d53de-03d4-4466-aaec-837de790e92a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:03.088527 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6d53de-03d4-4466-aaec-837de790e92a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:03.183634 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1453/2901] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:50:03 2022] POST /identity/v3/users => generated 287 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jun 10 13:50:03.187871 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7669d159-3b32-4777-a470-caf02501fbcd admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:03.198197 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7669d159-3b32-4777-a470-caf02501fbcd admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:03.204396 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab6bb1b-b10b-43e4-8309-c225a22c8ae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:03.205114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab6bb1b-b10b-43e4-8309-c225a22c8ae2 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:03.205399 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab6bb1b-b10b-43e4-8309-c225a22c8ae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:03.205675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab6bb1b-b10b-43e4-8309-c225a22c8ae2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:03.206407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab6bb1b-b10b-43e4-8309-c225a22c8ae2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:03.208009 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab6bb1b-b10b-43e4-8309-c225a22c8ae2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:03.218835 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1449/2902] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:50:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:50:03.222722 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6d53de-03d4-4466-aaec-837de790e92a admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:03.233464 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6d53de-03d4-4466-aaec-837de790e92a admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:03.238125 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941509f8-103f-41e9-b4ed-17c39fb0bc6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:03.238779 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941509f8-103f-41e9-b4ed-17c39fb0bc6a admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:03.239068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941509f8-103f-41e9-b4ed-17c39fb0bc6a admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:03.239351 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941509f8-103f-41e9-b4ed-17c39fb0bc6a admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:03.240711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941509f8-103f-41e9-b4ed-17c39fb0bc6a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:03.242369 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941509f8-103f-41e9-b4ed-17c39fb0bc6a admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:03.252418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1454/2903] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:50:03 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:50:03.260649 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab6bb1b-b10b-43e4-8309-c225a22c8ae2 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:03.270527 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab6bb1b-b10b-43e4-8309-c225a22c8ae2 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:03.274991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56cc297-48f2-4637-b0ff-0538cb14c2c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:03.275860 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56cc297-48f2-4637-b0ff-0538cb14c2c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:03.276130 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56cc297-48f2-4637-b0ff-0538cb14c2c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:03.276412 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56cc297-48f2-4637-b0ff-0538cb14c2c7 admin admin] PATH_INFO: `/v3/projects/80e9b2aa07844f95a4dde2aa8b037001/users/b88aa2912301493984a1bf0c7e1cc7ff/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:03.290994 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56cc297-48f2-4637-b0ff-0538cb14c2c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=80e9b2aa07844f95a4dde2aa8b037001, user_id=b88aa2912301493984a1bf0c7e1cc7ff, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:03.293758 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56cc297-48f2-4637-b0ff-0538cb14c2c7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:03.363215 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1450/2904] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:50:03 2022] PUT /identity/v3/projects/80e9b2aa07844f95a4dde2aa8b037001/users/b88aa2912301493984a1bf0c7e1cc7ff/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:03.375754 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7a0564-6d76-4ff0-81c8-7350de5ba6fe None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:03.376046 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7a0564-6d76-4ff0-81c8-7350de5ba6fe None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:03.376352 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca7a0564-6d76-4ff0-81c8-7350de5ba6fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:03.402217 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-ca7a0564-6d76-4ff0-81c8-7350de5ba6fe None None] MFA Rules not processed for user `b88aa2912301493984a1bf0c7e1cc7ff`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:03.403410 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-ca7a0564-6d76-4ff0-81c8-7350de5ba6fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:03.466388 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1455/2905] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:03 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:03.477819 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56cc297-48f2-4637-b0ff-0538cb14c2c7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:03.479639 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-a56cc297-48f2-4637-b0ff-0538cb14c2c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:03.534565 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56cc297-48f2-4637-b0ff-0538cb14c2c7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:03.539448 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e252d850-13bd-44f0-813f-4245e68a1e1a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:03.540085 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e252d850-13bd-44f0-813f-4245e68a1e1a service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:03.540366 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e252d850-13bd-44f0-813f-4245e68a1e1a service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:03.540642 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e252d850-13bd-44f0-813f-4245e68a1e1a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:03.548993 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e252d850-13bd-44f0-813f-4245e68a1e1a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:03.550419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e252d850-13bd-44f0-813f-4245e68a1e1a service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:03.561431 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1451/2906] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:50:03 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:50:07.437669 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7a0564-6d76-4ff0-81c8-7350de5ba6fe None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:07.504572 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca7a0564-6d76-4ff0-81c8-7350de5ba6fe None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:07.509272 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:07.510135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:07.510441 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:07.510711 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] PATH_INFO: `/v3/users/b88aa2912301493984a1bf0c7e1cc7ff` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:07.511883 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b88aa2912301493984a1bf0c7e1cc7ff)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:07.513928 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:07.699503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:07.705551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b88aa2912301493984a1bf0c7e1cc7ff'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:07.706203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b88aa2912301493984a1bf0c7e1cc7ff'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:07.706810 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b88aa2912301493984a1bf0c7e1cc7ff'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:07.714073 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b88aa2912301493984a1bf0c7e1cc7ff'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:07.803859 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b88aa2912301493984a1bf0c7e1cc7ff'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:07.827625 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1456/2907] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:50:07 2022] DELETE /identity/v3/users/b88aa2912301493984a1bf0c7e1cc7ff => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:07.833702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5babb395-9c7b-4b93-a73c-61f6384bbb87 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:07.834172 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5babb395-9c7b-4b93-a73c-61f6384bbb87 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:07.834271 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5babb395-9c7b-4b93-a73c-61f6384bbb87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:07.851503 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-5babb395-9c7b-4b93-a73c-61f6384bbb87 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:07.852702 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5babb395-9c7b-4b93-a73c-61f6384bbb87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:07.909643 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1452/2908] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:07.919135 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:07.920536 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:07.976415 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02273fc-0caa-4a6e-9a31-65ccd3e63d16 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:07.979993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c5ec91-d134-4a71-afb5-ba9cc0b2f86b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:07.980623 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c5ec91-d134-4a71-afb5-ba9cc0b2f86b service neutron] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:07.980942 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c5ec91-d134-4a71-afb5-ba9cc0b2f86b service neutron] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:07.981223 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c5ec91-d134-4a71-afb5-ba9cc0b2f86b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:07.990514 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c5ec91-d134-4a71-afb5-ba9cc0b2f86b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:07.991933 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0c5ec91-d134-4a71-afb5-ba9cc0b2f86b service neutron] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:08.001231 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1457/2909] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:50:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jun 10 13:50:08.393208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5babb395-9c7b-4b93-a73c-61f6384bbb87 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:08.394634 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5babb395-9c7b-4b93-a73c-61f6384bbb87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:08.401680 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:DELETE=5,keystone:SELECT=92,keystone:INSERT=3 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:08.406984 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5babb395-9c7b-4b93-a73c-61f6384bbb87 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:08.412214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:08.413104 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:08.413392 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:08.413690 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] PATH_INFO: `/v3/projects/80e9b2aa07844f95a4dde2aa8b037001` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:08.414883 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] RBAC: Authorizing `identity:delete_project(project_id=80e9b2aa07844f95a4dde2aa8b037001)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:08.416895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:08.532012 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '80e9b2aa07844f95a4dde2aa8b037001', 'request_id': 'req-c21a05f5-77b5-43f8-9a61-22a4f4739578'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:08.538778 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '80e9b2aa07844f95a4dde2aa8b037001', 'request_id': 'req-c21a05f5-77b5-43f8-9a61-22a4f4739578'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:08.544327 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '80e9b2aa07844f95a4dde2aa8b037001', 'request_id': 'req-c21a05f5-77b5-43f8-9a61-22a4f4739578'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:08.643118 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] The token cache is being invalidate because project 80e9b2aa07844f95a4dde2aa8b037001 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80300) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:50:08.643429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:08.644042 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-c21a05f5-77b5-43f8-9a61-22a4f4739578 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:08.645619 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1453/2910] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:50:08 2022] DELETE /identity/v3/projects/80e9b2aa07844f95a4dde2aa8b037001 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:08.650491 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16c9124-8499-45ed-ae1d-cf07d483f690 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:08.650983 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16c9124-8499-45ed-ae1d-cf07d483f690 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:08.651083 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d16c9124-8499-45ed-ae1d-cf07d483f690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:08.668903 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-d16c9124-8499-45ed-ae1d-cf07d483f690 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:08.670129 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-d16c9124-8499-45ed-ae1d-cf07d483f690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:08.728818 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1458/2911] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:08.736461 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e7296-2913-4a6f-92d3-ce7ec5c39103 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:08.736755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e7296-2913-4a6f-92d3-ce7ec5c39103 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:08.737057 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e7296-2913-4a6f-92d3-ce7ec5c39103 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:08.754675 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-9b4e7296-2913-4a6f-92d3-ce7ec5c39103 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:08.755873 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-9b4e7296-2913-4a6f-92d3-ce7ec5c39103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:08.763646 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1454/2912] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:08.767748 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16c9124-8499-45ed-ae1d-cf07d483f690 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:08.777857 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d16c9124-8499-45ed-ae1d-cf07d483f690 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:08.782443 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95ddda9-940f-44c1-b07d-325f8b411750 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:08.783035 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95ddda9-940f-44c1-b07d-325f8b411750 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:08.783319 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95ddda9-940f-44c1-b07d-325f8b411750 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:08.783595 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95ddda9-940f-44c1-b07d-325f8b411750 admin admin] PATH_INFO: `/v3/domains` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:08.784315 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95ddda9-940f-44c1-b07d-325f8b411750 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80299) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jun 10 13:50:08.784628 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95ddda9-940f-44c1-b07d-325f8b411750 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:08.786805 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95ddda9-940f-44c1-b07d-325f8b411750 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:08.798345 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1459/2913] 10.222.0.25 () {58 vars in 1152 bytes} [Fri Jun 10 13:50:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jun 10 13:50:08.804045 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e7296-2913-4a6f-92d3-ce7ec5c39103 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:08.814783 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e7296-2913-4a6f-92d3-ce7ec5c39103 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:08.819455 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150cbbdb-3d2f-478a-9fc5-f0f53ae77a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:08.820103 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150cbbdb-3d2f-478a-9fc5-f0f53ae77a5d admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:08.820390 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150cbbdb-3d2f-478a-9fc5-f0f53ae77a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:08.820671 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150cbbdb-3d2f-478a-9fc5-f0f53ae77a5d admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:08.821425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150cbbdb-3d2f-478a-9fc5-f0f53ae77a5d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:08.823398 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150cbbdb-3d2f-478a-9fc5-f0f53ae77a5d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:08.924740 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1455/2914] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:50:08 2022] POST /identity/v3/projects => generated 377 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:50:08.929160 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95ddda9-940f-44c1-b07d-325f8b411750 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:08.940357 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95ddda9-940f-44c1-b07d-325f8b411750 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:08.984095 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:08.996222 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae75d78-2ab9-4251-81fc-0013ea67fbd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:08.996923 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae75d78-2ab9-4251-81fc-0013ea67fbd0 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:08.997214 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae75d78-2ab9-4251-81fc-0013ea67fbd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:08.997503 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae75d78-2ab9-4251-81fc-0013ea67fbd0 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:08.998197 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae75d78-2ab9-4251-81fc-0013ea67fbd0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:09.000132 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae75d78-2ab9-4251-81fc-0013ea67fbd0 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:09.080678 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1460/2915] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:50:08 2022] POST /identity/v3/users => generated 289 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:50:09.084544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150cbbdb-3d2f-478a-9fc5-f0f53ae77a5d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:09.094893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150cbbdb-3d2f-478a-9fc5-f0f53ae77a5d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:09.099464 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e488be-f433-431e-af69-234ecd7b187a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:09.100107 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e488be-f433-431e-af69-234ecd7b187a admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.100389 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e488be-f433-431e-af69-234ecd7b187a admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.100669 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e488be-f433-431e-af69-234ecd7b187a admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.101469 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e488be-f433-431e-af69-234ecd7b187a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:09.103054 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e488be-f433-431e-af69-234ecd7b187a admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:09.113209 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1456/2916] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:50:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:50:09.117202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae75d78-2ab9-4251-81fc-0013ea67fbd0 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:09.127260 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae75d78-2ab9-4251-81fc-0013ea67fbd0 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:09.132625 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0e77d7-ffa1-4058-b160-8833275a8795 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:09.133311 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0e77d7-ffa1-4058-b160-8833275a8795 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.133594 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0e77d7-ffa1-4058-b160-8833275a8795 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.133870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0e77d7-ffa1-4058-b160-8833275a8795 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.134624 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0e77d7-ffa1-4058-b160-8833275a8795 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:09.136256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0e77d7-ffa1-4058-b160-8833275a8795 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:09.147960 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1461/2917] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:50:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:50:09.151904 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e488be-f433-431e-af69-234ecd7b187a admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:09.161590 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e488be-f433-431e-af69-234ecd7b187a admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:09.166114 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5618a-e8be-4c8c-87a8-03866a61e6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:09.166973 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5618a-e8be-4c8c-87a8-03866a61e6d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.167258 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5618a-e8be-4c8c-87a8-03866a61e6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.167534 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5618a-e8be-4c8c-87a8-03866a61e6d1 admin admin] PATH_INFO: `/v3/projects/8cbcda15807c431e9c7e436c1cc4a23f/users/0c5f983fb4ac4940a43af1ebd70dd363/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.181472 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5618a-e8be-4c8c-87a8-03866a61e6d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8cbcda15807c431e9c7e436c1cc4a23f, user_id=0c5f983fb4ac4940a43af1ebd70dd363, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:09.184119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5618a-e8be-4c8c-87a8-03866a61e6d1 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:09.257453 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1457/2918] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:50:09 2022] PUT /identity/v3/projects/8cbcda15807c431e9c7e436c1cc4a23f/users/0c5f983fb4ac4940a43af1ebd70dd363/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:09.267542 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b80b1b3-1516-449a-bda7-3c9cd8b6d717 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.267830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b80b1b3-1516-449a-bda7-3c9cd8b6d717 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.268105 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b80b1b3-1516-449a-bda7-3c9cd8b6d717 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.295777 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-7b80b1b3-1516-449a-bda7-3c9cd8b6d717 None None] MFA Rules not processed for user `0c5f983fb4ac4940a43af1ebd70dd363`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:09.297001 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-7b80b1b3-1516-449a-bda7-3c9cd8b6d717 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:09.360645 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1462/2919] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:09 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:09.365077 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5618a-e8be-4c8c-87a8-03866a61e6d1 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:09.424603 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5618a-e8be-4c8c-87a8-03866a61e6d1 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:09.429732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eaf5418-7de8-4463-a70a-12a61b0f3ce2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:09.430377 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaf5418-7de8-4463-a70a-12a61b0f3ce2 admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.430661 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaf5418-7de8-4463-a70a-12a61b0f3ce2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.430941 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eaf5418-7de8-4463-a70a-12a61b0f3ce2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.431628 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eaf5418-7de8-4463-a70a-12a61b0f3ce2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:09.433560 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6eaf5418-7de8-4463-a70a-12a61b0f3ce2 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:09.526139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1458/2920] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:50:09 2022] POST /identity/v3/projects => generated 375 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:50:09.530133 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b80b1b3-1516-449a-bda7-3c9cd8b6d717 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:09.541032 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b80b1b3-1516-449a-bda7-3c9cd8b6d717 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:09.596371 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9892cd02-37c0-439a-a4f5-229a8c930084 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:09.596993 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9892cd02-37c0-439a-a4f5-229a8c930084 admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.597282 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9892cd02-37c0-439a-a4f5-229a8c930084 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.597563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9892cd02-37c0-439a-a4f5-229a8c930084 admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.598250 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9892cd02-37c0-439a-a4f5-229a8c930084 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:09.600177 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9892cd02-37c0-439a-a4f5-229a8c930084 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:09.717571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1463/2921] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:50:09 2022] POST /identity/v3/users => generated 288 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:50:09.721277 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eaf5418-7de8-4463-a70a-12a61b0f3ce2 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:09.731470 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eaf5418-7de8-4463-a70a-12a61b0f3ce2 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:09.736214 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d743400c-fbcc-4ab1-b698-09f42d29e3de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:09.736938 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d743400c-fbcc-4ab1-b698-09f42d29e3de admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.737252 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d743400c-fbcc-4ab1-b698-09f42d29e3de admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.737530 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d743400c-fbcc-4ab1-b698-09f42d29e3de admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.738260 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d743400c-fbcc-4ab1-b698-09f42d29e3de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:09.739970 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d743400c-fbcc-4ab1-b698-09f42d29e3de admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:09.741755 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:09.752112 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1459/2922] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:50:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:50:09.756375 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9892cd02-37c0-439a-a4f5-229a8c930084 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:09.766377 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9892cd02-37c0-439a-a4f5-229a8c930084 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:09.771120 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c45899-980c-47fc-88f0-0772172d0bf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:09.771858 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c45899-980c-47fc-88f0-0772172d0bf7 admin admin] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.772142 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c45899-980c-47fc-88f0-0772172d0bf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.772515 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c45899-980c-47fc-88f0-0772172d0bf7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.773263 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c45899-980c-47fc-88f0-0772172d0bf7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:09.775024 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c45899-980c-47fc-88f0-0772172d0bf7 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:09.784891 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1464/2923] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:50:09 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:50:09.789082 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d743400c-fbcc-4ab1-b698-09f42d29e3de admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:09.798782 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d743400c-fbcc-4ab1-b698-09f42d29e3de admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:09.803814 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c3abe4-ea45-4270-8689-5c81cffda846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:09.804686 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c3abe4-ea45-4270-8689-5c81cffda846 admin admin] REQUEST_METHOD: `PUT` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.804996 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c3abe4-ea45-4270-8689-5c81cffda846 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.805267 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c3abe4-ea45-4270-8689-5c81cffda846 admin admin] PATH_INFO: `/v3/projects/08ba61cc51c940f5bd945070155053e0/users/20d20b8042b64a9187779ab01ecfdf77/roles/dfc4731db42b4f0bbb5e10d84814f9cd` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.820111 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c3abe4-ea45-4270-8689-5c81cffda846 admin admin] RBAC: Authorizing `identity:create_grant(project_id=08ba61cc51c940f5bd945070155053e0, user_id=20d20b8042b64a9187779ab01ecfdf77, role_id=dfc4731db42b4f0bbb5e10d84814f9cd)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:09.822891 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c3abe4-ea45-4270-8689-5c81cffda846 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:09.922827 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1460/2924] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:50:09 2022] PUT /identity/v3/projects/08ba61cc51c940f5bd945070155053e0/users/20d20b8042b64a9187779ab01ecfdf77/roles/dfc4731db42b4f0bbb5e10d84814f9cd => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jun 10 13:50:09.934033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfdffbf-0f8c-47b8-b761-c14bb15d15f4 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:09.934332 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfdffbf-0f8c-47b8-b761-c14bb15d15f4 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:09.934666 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfdffbf-0f8c-47b8-b761-c14bb15d15f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:09.961563 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-4cfdffbf-0f8c-47b8-b761-c14bb15d15f4 None None] MFA Rules not processed for user `20d20b8042b64a9187779ab01ecfdf77`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:09.962770 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-4cfdffbf-0f8c-47b8-b761-c14bb15d15f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:09.981870 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:10.025053 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1465/2925] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:09 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:10.029379 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c3abe4-ea45-4270-8689-5c81cffda846 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:10.088997 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c3abe4-ea45-4270-8689-5c81cffda846 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:10.093514 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6302b215-e655-4bca-badd-1ae4ee97577b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:10.094151 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6302b215-e655-4bca-badd-1ae4ee97577b admin admin] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:10.094429 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6302b215-e655-4bca-badd-1ae4ee97577b admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:10.094703 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6302b215-e655-4bca-badd-1ae4ee97577b admin admin] PATH_INFO: `/v3/projects` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:10.095374 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6302b215-e655-4bca-badd-1ae4ee97577b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:10.097295 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6302b215-e655-4bca-badd-1ae4ee97577b admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:10.217893 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1461/2926] 10.222.0.25 () {60 vars in 1152 bytes} [Fri Jun 10 13:50:10 2022] POST /identity/v3/projects => generated 375 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:50:10.221991 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfdffbf-0f8c-47b8-b761-c14bb15d15f4 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:10.234615 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cfdffbf-0f8c-47b8-b761-c14bb15d15f4 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:10.289893 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3359f905-a76e-424b-bc0c-b473ababbc9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:10.290562 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3359f905-a76e-424b-bc0c-b473ababbc9e admin admin] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:10.290845 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3359f905-a76e-424b-bc0c-b473ababbc9e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:10.291122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3359f905-a76e-424b-bc0c-b473ababbc9e admin admin] PATH_INFO: `/v3/users` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:10.291792 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3359f905-a76e-424b-bc0c-b473ababbc9e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:10.293746 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3359f905-a76e-424b-bc0c-b473ababbc9e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:10.369418 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1466/2927] 10.222.0.25 () {60 vars in 1143 bytes} [Fri Jun 10 13:50:10 2022] POST /identity/v3/users => generated 294 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jun 10 13:50:10.373383 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6302b215-e655-4bca-badd-1ae4ee97577b admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:10.384210 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6302b215-e655-4bca-badd-1ae4ee97577b admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:10.388720 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeca183c-c158-45d6-a818-5f108a6c3ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:10.389408 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeca183c-c158-45d6-a818-5f108a6c3ad7 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:10.389691 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeca183c-c158-45d6-a818-5f108a6c3ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:10.389967 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeca183c-c158-45d6-a818-5f108a6c3ad7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:10.390732 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeca183c-c158-45d6-a818-5f108a6c3ad7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:10.392314 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeca183c-c158-45d6-a818-5f108a6c3ad7 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:10.402630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1462/2928] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:50:10 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:50:10.406411 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3359f905-a76e-424b-bc0c-b473ababbc9e admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:10.416534 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3359f905-a76e-424b-bc0c-b473ababbc9e admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:10.421068 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5811256a-fb1c-4109-a088-01b2c53f88c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:10.421984 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5811256a-fb1c-4109-a088-01b2c53f88c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:10.422264 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5811256a-fb1c-4109-a088-01b2c53f88c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:10.422539 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5811256a-fb1c-4109-a088-01b2c53f88c3 admin admin] PATH_INFO: `/v3/projects/6ef2f3db7a4b4a3292569efef50e8b86/users/3831a6eccde745ccac92bb367c28110d/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:10.437280 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5811256a-fb1c-4109-a088-01b2c53f88c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ef2f3db7a4b4a3292569efef50e8b86, user_id=3831a6eccde745ccac92bb367c28110d, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:10.439924 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5811256a-fb1c-4109-a088-01b2c53f88c3 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:10.534635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1467/2929] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:50:10 2022] PUT /identity/v3/projects/6ef2f3db7a4b4a3292569efef50e8b86/users/3831a6eccde745ccac92bb367c28110d/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:10.538518 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeca183c-c158-45d6-a818-5f108a6c3ad7 admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:10.548419 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeca183c-c158-45d6-a818-5f108a6c3ad7 admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:10.603264 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073ba256-1a9c-42e9-8c3c-5572bf7e5956 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:10.603912 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073ba256-1a9c-42e9-8c3c-5572bf7e5956 admin admin] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:10.604191 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073ba256-1a9c-42e9-8c3c-5572bf7e5956 admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:10.604468 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073ba256-1a9c-42e9-8c3c-5572bf7e5956 admin admin] PATH_INFO: `/v3/roles` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:10.605211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073ba256-1a9c-42e9-8c3c-5572bf7e5956 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:10.606794 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073ba256-1a9c-42e9-8c3c-5572bf7e5956 admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:10.616821 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1463/2930] 10.222.0.25 () {58 vars in 1121 bytes} [Fri Jun 10 13:50:10 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jun 10 13:50:10.620887 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5811256a-fb1c-4109-a088-01b2c53f88c3 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:10.631128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5811256a-fb1c-4109-a088-01b2c53f88c3 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:10.635707 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b34e20a-ba60-4d9a-b78f-880684b30f54 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:10.636571 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b34e20a-ba60-4d9a-b78f-880684b30f54 admin admin] REQUEST_METHOD: `PUT` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:10.636889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b34e20a-ba60-4d9a-b78f-880684b30f54 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:10.637172 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b34e20a-ba60-4d9a-b78f-880684b30f54 admin admin] PATH_INFO: `/v3/projects/6ef2f3db7a4b4a3292569efef50e8b86/users/3831a6eccde745ccac92bb367c28110d/roles/766799d1b6c049168590571f1a959b3b` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:10.639236 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b34e20a-ba60-4d9a-b78f-880684b30f54 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ef2f3db7a4b4a3292569efef50e8b86, user_id=3831a6eccde745ccac92bb367c28110d, role_id=766799d1b6c049168590571f1a959b3b)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:10.642031 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b34e20a-ba60-4d9a-b78f-880684b30f54 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:10.652564 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1468/2931] 10.222.0.25 () {60 vars in 1482 bytes} [Fri Jun 10 13:50:10 2022] PUT /identity/v3/projects/6ef2f3db7a4b4a3292569efef50e8b86/users/3831a6eccde745ccac92bb367c28110d/roles/766799d1b6c049168590571f1a959b3b => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:10.663099 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0843735f-fbe3-40e1-8722-e38ba7aad966 None None] REQUEST_METHOD: `POST` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:10.666983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0843735f-fbe3-40e1-8722-e38ba7aad966 None None] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:10.666983 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0843735f-fbe3-40e1-8722-e38ba7aad966 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:10.690839 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.auth.core [None req-0843735f-fbe3-40e1-8722-e38ba7aad966 None None] MFA Rules not processed for user `3831a6eccde745ccac92bb367c28110d`. Rule list: `[]` (Enabled: `True`). {{(pid=80300) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:10.692074 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0843735f-fbe3-40e1-8722-e38ba7aad966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:10.720630 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:10.758597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1464/2932] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:10 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:10.769530 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b34e20a-ba60-4d9a-b78f-880684b30f54 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:10.771246 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-0b34e20a-ba60-4d9a-b78f-880684b30f54 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:10.833241 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b34e20a-ba60-4d9a-b78f-880684b30f54 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:10.836830 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280ae8ed-95a3-4ca5-be48-6ba05edf9b96 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:10.837477 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280ae8ed-95a3-4ca5-be48-6ba05edf9b96 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:10.837760 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280ae8ed-95a3-4ca5-be48-6ba05edf9b96 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:10.838037 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280ae8ed-95a3-4ca5-be48-6ba05edf9b96 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:10.867635 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:10.894455 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280ae8ed-95a3-4ca5-be48-6ba05edf9b96 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:10.895902 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-280ae8ed-95a3-4ca5-be48-6ba05edf9b96 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:10.904642 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1469/2933] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:50:10 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:50:12.821539 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0843735f-fbe3-40e1-8722-e38ba7aad966 None None] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:12.822955 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-0843735f-fbe3-40e1-8722-e38ba7aad966 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:12.835140 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0843735f-fbe3-40e1-8722-e38ba7aad966 None None] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:12.839213 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5080b8d-0174-40f6-a193-390137b9c6b9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:12.839841 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5080b8d-0174-40f6-a193-390137b9c6b9 service cinder] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:12.840119 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5080b8d-0174-40f6-a193-390137b9c6b9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:12.840394 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5080b8d-0174-40f6-a193-390137b9c6b9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:12.894159 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5080b8d-0174-40f6-a193-390137b9c6b9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:12.895589 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5080b8d-0174-40f6-a193-390137b9c6b9 service cinder] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:12.904783 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1465/2934] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:50:12 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:50:14.428506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280ae8ed-95a3-4ca5-be48-6ba05edf9b96 service cinder] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:14.437752 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-280ae8ed-95a3-4ca5-be48-6ba05edf9b96 service cinder] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:14.441380 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de57f7e1-19f4-4de3-b7bc-01807192c7f3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '184bf14f33464130ab62ed1853bf4f8d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:14.442022 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de57f7e1-19f4-4de3-b7bc-01807192c7f3 service cinder] REQUEST_METHOD: `GET` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:14.442309 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de57f7e1-19f4-4de3-b7bc-01807192c7f3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:14.442591 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de57f7e1-19f4-4de3-b7bc-01807192c7f3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:14.452040 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de57f7e1-19f4-4de3-b7bc-01807192c7f3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:14.453517 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de57f7e1-19f4-4de3-b7bc-01807192c7f3 service cinder] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:14.463186 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1470/2935] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:50:14 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:50:19.055287 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5080b8d-0174-40f6-a193-390137b9c6b9 service cinder] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:19.115964 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5080b8d-0174-40f6-a193-390137b9c6b9 service cinder] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:19.120914 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:19.121792 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:19.122072 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:19.122344 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] PATH_INFO: `/v3/users/0c5f983fb4ac4940a43af1ebd70dd363` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:19.123516 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c5f983fb4ac4940a43af1ebd70dd363)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:19.125456 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:19.308139 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c5f983fb4ac4940a43af1ebd70dd363'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:19.308711 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c5f983fb4ac4940a43af1ebd70dd363'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:19.315572 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c5f983fb4ac4940a43af1ebd70dd363'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:19.397597 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c5f983fb4ac4940a43af1ebd70dd363'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:19.398211 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c5f983fb4ac4940a43af1ebd70dd363'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:19.403037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:19.415906 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1466/2936] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:50:19 2022] DELETE /identity/v3/users/0c5f983fb4ac4940a43af1ebd70dd363 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:19.420506 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b41902-d8b4-4f43-995e-046f165680a7 None None] REQUEST_METHOD: `POST` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:19.421033 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b41902-d8b4-4f43-995e-046f165680a7 None None] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:19.421162 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b41902-d8b4-4f43-995e-046f165680a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:19.438613 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.auth.core [None req-95b41902-d8b4-4f43-995e-046f165680a7 None None] MFA Rules not processed for user `05ecde7b5ffd440ea8bdbb9c0c6418c1`. Rule list: `[]` (Enabled: `True`). {{(pid=80299) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jun 10 13:50:19.439849 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-95b41902-d8b4-4f43-995e-046f165680a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:19.498972 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1471/2937] 10.222.0.25 () {58 vars in 957 bytes} [Fri Jun 10 13:50:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jun 10 13:50:19.508425 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:19.509847 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:19.567014 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-709b4910-3143-4efc-ae30-4001135636cf admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:19.570564 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86de5832-db22-45f9-8fb5-8cb13ee738fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6794257c90e40bca2c8efd31b0c791a', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:19.571206 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86de5832-db22-45f9-8fb5-8cb13ee738fd service neutron] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:19.571490 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86de5832-db22-45f9-8fb5-8cb13ee738fd service neutron] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:19.571770 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86de5832-db22-45f9-8fb5-8cb13ee738fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:19.580895 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86de5832-db22-45f9-8fb5-8cb13ee738fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:19.582335 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86de5832-db22-45f9-8fb5-8cb13ee738fd service neutron] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:19.591623 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1467/2938] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:50:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:50:20.224256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b41902-d8b4-4f43-995e-046f165680a7 None None] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:20.227146 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-95b41902-d8b4-4f43-995e-046f165680a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:20.245916 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b41902-d8b4-4f43-995e-046f165680a7 None None] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:20.250507 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:20.251376 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:20.251663 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:20.252069 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] PATH_INFO: `/v3/projects/8cbcda15807c431e9c7e436c1cc4a23f` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:20.253185 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8cbcda15807c431e9c7e436c1cc4a23f)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:20.255166 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:20.359943 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8cbcda15807c431e9c7e436c1cc4a23f', 'request_id': 'req-1f9f5088-c758-4685-baf6-274a19351cb5'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.428254 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8cbcda15807c431e9c7e436c1cc4a23f', 'request_id': 'req-1f9f5088-c758-4685-baf6-274a19351cb5'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.436122 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8cbcda15807c431e9c7e436c1cc4a23f', 'request_id': 'req-1f9f5088-c758-4685-baf6-274a19351cb5'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.446284 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] The token cache is being invalidate because project 8cbcda15807c431e9c7e436c1cc4a23f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:50:20.446573 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.447202 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.448728 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1472/2939] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:50:20 2022] DELETE /identity/v3/projects/8cbcda15807c431e9c7e436c1cc4a23f => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:20.453038 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86de5832-db22-45f9-8fb5-8cb13ee738fd service neutron] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:20.462407 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86de5832-db22-45f9-8fb5-8cb13ee738fd service neutron] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:20.517955 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:20.518917 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:20.519202 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:20.519480 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] PATH_INFO: `/v3/users/20d20b8042b64a9187779ab01ecfdf77` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:20.520751 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] RBAC: Authorizing `identity:delete_user(user_id=20d20b8042b64a9187779ab01ecfdf77)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:20.523166 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:20.724918 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20d20b8042b64a9187779ab01ecfdf77'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.725573 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20d20b8042b64a9187779ab01ecfdf77'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.732324 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20d20b8042b64a9187779ab01ecfdf77'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.803239 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20d20b8042b64a9187779ab01ecfdf77'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.803859 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20d20b8042b64a9187779ab01ecfdf77'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:20.822238 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1468/2940] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:50:20 2022] DELETE /identity/v3/users/20d20b8042b64a9187779ab01ecfdf77 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:21.223912 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:21.228205 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:21.239622 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9f5088-c758-4685-baf6-274a19351cb5 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:21.277630 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:21.295807 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:21.296662 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:21.296976 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:21.297256 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] PATH_INFO: `/v3/projects/08ba61cc51c940f5bd945070155053e0` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:21.298417 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=08ba61cc51c940f5bd945070155053e0)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:21.300366 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:21.408824 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08ba61cc51c940f5bd945070155053e0', 'request_id': 'req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.465736 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08ba61cc51c940f5bd945070155053e0', 'request_id': 'req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.472200 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08ba61cc51c940f5bd945070155053e0', 'request_id': 'req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.483889 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] The token cache is being invalidate because project 08ba61cc51c940f5bd945070155053e0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:50:21.484219 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.484938 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.486414 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1473/2941] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:50:21 2022] DELETE /identity/v3/projects/08ba61cc51c940f5bd945070155053e0 => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:21.490371 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:21.491746 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:21.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8256fa-e895-4bc4-9f61-1d697b3a9c6d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:21.507484 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:21.559343 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:21.560208 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:21.560495 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:21.560796 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] PATH_INFO: `/v3/users/3831a6eccde745ccac92bb367c28110d` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:21.562037 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3831a6eccde745ccac92bb367c28110d)` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:21.563957 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:21.771116 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3831a6eccde745ccac92bb367c28110d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.771854 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3831a6eccde745ccac92bb367c28110d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.779607 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3831a6eccde745ccac92bb367c28110d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.845991 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3831a6eccde745ccac92bb367c28110d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.846615 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.notifications [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3831a6eccde745ccac92bb367c28110d'} {{(pid=80300) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:21.868725 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1469/2942] 10.222.0.25 () {58 vars in 1223 bytes} [Fri Jun 10 13:50:21 2022] DELETE /identity/v3/users/3831a6eccde745ccac92bb367c28110d => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:22.286692 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] Authenticating user token {{(pid=80299) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:50:22.288391 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.fernet_utils [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80299) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:50:22.301128 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47adc1ee-a23a-4a41-b2a9-3b8e487084e7 admin admin] Validating token access rules against request {{(pid=80299) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:50:22.359353 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '05ecde7b5ffd440ea8bdbb9c0c6418c1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '96340edb86a840a88d51d74872a27bac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80299) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:50:22.360208 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:50:22.360501 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] SCRIPT_NAME: `/identity` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:50:22.360790 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] PATH_INFO: `/v3/projects/6ef2f3db7a4b4a3292569efef50e8b86` {{(pid=80299) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:50:22.361973 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ef2f3db7a4b4a3292569efef50e8b86)` {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:50:22.363905 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] RBAC: Authorization granted {{(pid=80299) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:50:22.477203 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ef2f3db7a4b4a3292569efef50e8b86', 'request_id': 'req-8eafc8bb-5572-46a7-9e54-e3e09868235e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:22.540420 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ef2f3db7a4b4a3292569efef50e8b86', 'request_id': 'req-8eafc8bb-5572-46a7-9e54-e3e09868235e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:22.547551 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ef2f3db7a4b4a3292569efef50e8b86', 'request_id': 'req-8eafc8bb-5572-46a7-9e54-e3e09868235e'} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:22.554771 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG dbcounter [-] [80299] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80299) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:50:22.559004 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] The token cache is being invalidate because project 6ef2f3db7a4b4a3292569efef50e8b86 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80299) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jun 10 13:50:22.559289 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:22.559911 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: DEBUG keystone.notifications [None req-8eafc8bb-5572-46a7-9e54-e3e09868235e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80299) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jun 10 13:50:22.561485 dv-cinder-master-3-iscsi devstack@keystone.service[80299]: [pid: 80299|app: 0|req: 1474/2943] 10.222.0.25 () {58 vars in 1232 bytes} [Fri Jun 10 13:50:22 2022] DELETE /identity/v3/projects/6ef2f3db7a4b4a3292569efef50e8b86 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jun 10 13:50:31.864817 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=50,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jun 10 13:54:48.441435 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] Authenticating user token {{(pid=80300) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jun 10 13:54:48.442864 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:54:48.531741 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65cd12ea-15c9-48c7-9d96-7fc4dd57ac4d admin admin] Validating token access rules against request {{(pid=80300) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jun 10 13:54:48.595882 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-118f1cda-e9eb-438c-bc1b-c792e5f6e334 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '0c315cfb1aa44f3c9a755f3cb8e5cd5d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c1103ec5f93b4ee1a5d72cff4c40178a', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80300) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jun 10 13:54:48.596465 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118f1cda-e9eb-438c-bc1b-c792e5f6e334 service placement] REQUEST_METHOD: `GET` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jun 10 13:54:48.596752 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118f1cda-e9eb-438c-bc1b-c792e5f6e334 service placement] SCRIPT_NAME: `/identity` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jun 10 13:54:48.597064 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.server.flask.request_processing.req_logging [None req-118f1cda-e9eb-438c-bc1b-c792e5f6e334 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80300) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jun 10 13:54:48.598857 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.fernet_utils [None req-118f1cda-e9eb-438c-bc1b-c792e5f6e334 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80300) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jun 10 13:54:48.610210 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118f1cda-e9eb-438c-bc1b-c792e5f6e334 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jun 10 13:54:48.615544 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-118f1cda-e9eb-438c-bc1b-c792e5f6e334 service placement] RBAC: Authorization granted {{(pid=80300) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jun 10 13:54:48.622158 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: [pid: 80300|app: 0|req: 1470/2944] 10.222.0.25 () {60 vars in 1365 bytes} [Fri Jun 10 13:54:48 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jun 10 13:54:58.618558 dv-cinder-master-3-iscsi devstack@keystone.service[80300]: DEBUG dbcounter [-] [80300] Writing DB stats keystone:SELECT=68 {{(pid=80300) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}